A > B, B > A, Sites Basics
In case that you have 2 or more geo locations, infrastructure can look something like this:
[___Internet___]
_____________________________________________________ | _____________________________________________________
| [ an-node01 ] | | | [ an-node02 ] |
| ____________ ______________| ____|____ |______________ ____________ |
| | vbr2 |--| bond2 | | [ IFN ] | | bond2 |--| vbr2 | |
| _________________ | 10.255.0.1 | | ______ | _|_________|_ | ______ | | 10.255.0.2 | ................... |
| | [ vm0001-dev ] | |____________| || eth2 =--\ | | Switch 1 | | /--= eth2 || |____________| : [ vm0001-dev ] : |
| | [ Dev Server ] | | | : : ||_____| \--=-----|_____________|-----=--/ |_____|| | | : : : [ Dev Server ] : |
| | ______| | | : : | ______ /--=-----| Switch 2 |-----=--\ ______ | | | : : :....... : |
| | | eth0 =----/ | : : || eth5 =--/ | |_____________| | \--= eth5 || | | : :----= eth0 : : |
| | |_____|| | : : ||_____| | | |_____|| | | : ::.....: : |
| | 10.254.0.1 | | : : |______________| |______________| | | : : : |
| |_________________| | : : ______________| |______________ | | : :.................: |
| | : : | bond1 | _________ | bond1 | | | : |
| _________________ | : : | 10.10.0.1 | | [ SN ] | | 10.10.0.2 | | | : ................... |
| | [ vm0002-web ] | | : : | ______ | _|_________|_ | ______ | | | : : [ vm0002-web ] : |
| | [ Web Server ] | | : : || eth1 =--\ | | Switch 1 | | /--= eth1 || | | : : [ Web Server ] : |
| | ______| | : : ||_____| \--=-----|_____________|-----=--/ |_____|| | | : :....... : |
| | | eth0 =------/ : : | ______ /--=-----| Switch 2 |-----=--\ ______ | | | :------= eth0 : : |
| | |_____|| : : || eth4 =--/ | |_____________| | \--= eth4 || | | ::.....: : |
| | 10.254.0.2 | : : ||_____| | | |_____|| | | : : |
| |_________________| : : |______________| |______________| | | :.................: |
| : : ______________| |______________ | | |
| ................... : : | bond0 | _________ | bond0 | | | _________________ |
| : [ vm0003-db ] : : : | 10.20.0.1 | | [ BCN ] | | 10.20.0.2 | | | | [ vm0003-db ] | |
| : [ DB Server ] : : : | ______ | _|_________|_ | ______ | | | | [ DB Server ] | |
| : .......: : : || eth0 =--\ | /--| Switch 1 |--\ | /--= eth0 || | | |______ | |
| : : eth0 =--------: : ||_____| \--=--+--|_____________|--+--=--/ |_____|| | \--------= eth0 | | |
| : :.....:: : | ______ /--=--+--| Switch 2 |--+--=--\ ______ | | ||_____| | |
| : : : || eth3 =--/ | | |_____________| | | \--= eth3 || | | 10.254.0.3 | |
| :.................: : ||_____| | | | | | | |_____|| | |_________________| |
| : |______________| | | | | |______________| | |
| ................... : | | | | | | | _________________ |
| : [ vm0004-win ] : : | | | | | | | | [ vm0004-win ] | |
| : [ MS Server ] : : | | | | | | | | [ MS Server ] | |
| : .......: : | | | | | | | |______ | |
| : : NIC0 =----------: | | | | | | \----------= NIC0 | | |
| : :.....:: ______| | | | | |______ ||_____| | |
| : : _____ | IPMI =--/ | | \--= IPMI | _____ | 10.254.0.4 | |
| :.................: [_BMC_]--|_____|| | | ||_____|--[_BMC_] |_________________| |
| | | | | |
| ______ ______ | | | | ______ ______ |
| | PSU1 | PSU2 | | | | | | PSU2 | PSU1 | |
|________________________________|______|______|______| | | |______|______|______|________________________________|
|| || ____|_ _|____ || ||
|| || | PDU1 | | PDU2 | || ||
|| || |______| |______| || ||
|| || || || || || || ||
|| \\===[ Power 1 ]===// || || \\===[ Power 1 ]===// ||
\\======[ Power 2 ]======||=====// ||
\\=============[ Power 2 ]======//
#### Storage part
_________________________________________________ _________________________________________________
| [ an-node01 ] | | [ an-node02 ] |
| ________ __________ | | __________ ________ |
| [_disk_1_]--+--[_/dev/sda_] | | [_/dev/sda_]--+--[_disk_1_] |
| ________ | | ___________ _______ | | _______ ___________ | | ________ |
| [_disk_2_]--+ +--[_/dev/sda1_]--[_/boot_] | | [_/boot_]--[_/dev/sda1_]--+ +--[_disk_2_] |
| ________ | | ___________ _______ | | ________ ___________ | | ________ |
| [_disk_3_]--+ +--[_/dev/sda2_]--[__swap_] | | [__swap__]--[_/dev/sda2_]--+ +--[_disk_3_] |
| ________ | | ___________ ___ | | ___ ___________ | | ________ |
| [_disk_4_]--/ +--[_/dev/sda3_]--[_/_] | | [_/_]--[_/dev/sda3_]--+ \--[_disk_4_] |
| | ___________ | | ___________ | |
| +--[_/dev/sda5_]------------\ | | /------------[_/dev/sda5_]--+ |
| | ___________ | | | | ___________ | |
| +--[_/dev/sda6_]----------\ | | | | /----------[_/dev/sda6_]--+ |
| | ___________ | | | | | | ___________ | |
| \--[_/dev/sda7_]--------\ | | | | | | /--------[_/dev/sda7_]--/ |
| _______________ ____________ | | | | | | | | ____________ _______________ |
| /--[_Clustered_LVM_]--[_/dev/drbd2_]--/ | | | | | | \--[_/dev/drbd2_]--[_Clustered_LVM_]--\ |
| _|__ | _______ | | | | | | | _______ __|_ |
| [_PV_] \--{_bond1_} | | | | | | \--{_bond1_} [_PV_] |
| _|________ | | | | | | ________|_ |
| [_an02-vg0_] | | | | | | [_an02-vg0_] |
| | ________________________ ....... | | | | | | _____ ________________________ | |
| +--[_/dev/an02-vg0/vm0003_1_]---:.vm3.: | | | | | | [_vm3_]---[_/dev/an02-vg0/vm0003_1_]--+ |
| | ________________________ ....... | | | | | | _____ ________________________ | |
| \--[_/dev/an02-vg0/vm0004_1_]---:.vm4.: | | | | | | [_vm4_]---[_/dev/an02-vg0/vm0004_1_]--/ |
| _______________ ____________ | | | | | | ____________ _______________ |
| /--[_Clustered_LVM_]--[_/dev/drbd1_]--/ | | | | \--[_/dev/drbd1_]--[_Clustered_LVM_]--\ |
| _|__ | _______ | | | | | _______ __|_ |
| [_PV_] \--{_bond1_} | | | | \--{_bond1_} [_PV_] |
| _|________ | | | | ________|_ |
| [_an01-vg0_] | | | | [_an01-vg0_] |
| | ________________________ _____ | | | | ....... ________________________ | |
| +--[_/dev/an01-vg0/vm0001_1_]---[_vm1_] | | | | :.vm1.:---[_/dev/an01-vg0/vm0001_1_]--+ |
| | ________________________ _____ | | | | ....... ________________________ | |
| \--[_/dev/an01-vg0/vm0002_1_]---[_vm2_] | | | | :.vm2.:---[_/dev/an01-vg0/vm0002_1_]--/ |
| _______________ ____________ | | | | ____________ _______________ |
| /--[_Clustered_LVM_]--[_/dev/drbd0_]--/ | | \--[_/dev/drbd0_]--[_Clustered_LVM_]--\ |
| _|__ | _______ | | | _______ __|_ |
| [_PV_] \--{_bond1_} | | \--{_bond1_} [_PV_] |
| _|__________ | | __________|_ |
| [_shared-vg0_] | | [_shared-vg0_] |
| _|_________________________ | | _________________________|_ |
| [_/dev/shared-vg0/lv_shared_] | | [_/dev/shared-vg0/lv_shared_] |
| | ______ _________ | | _________ ______ | |
| \--[_GFS2_]--[_/shared_] | | [_/shared_]--[_GFS2_]--/ |
| _______| _________ |_______ |
| | bond1 =--| Storage |--= bond1 | |
| |______|| | Network | ||______| |
|_________________________________________________| |_________| |_________________________________________________|
Locks handling using CPG
- The cluster starts up cleanly with two members.
- Both members are able to start service:foo.
- Both want to start it, but need a lock from DLM to do so.
- The an-node01 member has its totem token, and sends its request for the lock.
- DLM issues a lock for that service to an-node01.
- The an-node02 member requests a lock for the same service.
- DLM rejects the lock request.
- The an-node01 member successfully starts service:foo and announces this to the CPG members.
- The an-node02 sees that service:foo is now running on an-node01 and no longer tries to start the service.
Storage part
- The two members want to write to a common area of the /shared GFS2 partition.
- The an-node02 sends a request for a DLM lock against the FS, gets it.
- The an-node01 sends a request for the same lock, but DLM sees that a lock is pending and rejects the request.
- The an-node02 member finishes altering the file system, announces the change over CPG and releases the lock.
- The an-node01 member updates its view of the filesystem, requests a lock, receives it and proceeds to update the filesystems.
- It completes the changes, announces the changes over CPG and releases the lock.
Messages can only be sent to the members of the CPG while the node has a totem token from corosync.