I'm migrating some of our aging Xen 2.x servers to Xen 3 in a SAN environment. We've got two IBM e326 servers attached to a FibreChannel SAN which will host the guest file systems as LVM logical volumes.
At first, I though I'd need some form of clustering in order to arbitrate access to the SAN. Turns out CLVM is overkill for this type of thing. Instead, I'm relying on a sparingly used feature of LVM2; host tags.
Nate Carlson asked about this exact issue with Xen hosts on the linux-cluster mailing list back in November. His question wasn't directly answered on the list, so here goes.
The idea is that both physical machines will "see" the logical LVM block devices, but only one machine at a time will actually activate the logical volumes. Once this is all up and running, I plan to investigate the case of live migration, where the guest operating system moves from one physical machine to another without service interruption. In this case, I'll have to active the logiocal volume on both Xen hosts.
In the meantime, here's how I've got it setup. Check out Appendix C of the Cluster Logical Volume Manager Guide.
I added the following lines to the standard lvm configuration file on both physical Xen hosts.
# /etc/lvm/lvm.conf
tags { hosttags = 1 }
volume_list = [ "VolGroup00", "@*" ]
Using LVM tags tells each physical machine which logical volumes it may and may not activate.
My test SAN volume is /dev/sdb on both physical machines (xen02 and xen03). I created a single physical volume both machines will use. There's one volume group for all xen guest file systems.
# Execute on either physical xen host
pvcreate /dev/sdb1
vgcreate xensan /dev/sdb1
lvcreate --addtag $(uname -n) -L 8192M -n ldap1.rootfs xensan
lvcreate --addtag $(uname -n) -L 1024M -n ldap1.swapfs xensan
NOTE: I don't prefix the value of the --addtag option with an @ like the linked manual says to, I found this didn't work as expected.
Now we have logical volumes created on the SAN. Both physical xen hosts will see these physical volumes, but tagging prevents them from clobbering each other:
Remember, the volumes are tagged with "xen03.math.ohio-state.edu"
[root@xen02 ~]# vgchange -ay
0 logical volume(s) in volume group "xensan" now active
2 logical volume(s) in volume group "VolGroup00" now active
[root@xen03 ~]# vgchange -ay
2 logical volume(s) in volume group "xensan" now active
2 logical volume(s) in volume group "VolGroup00" now active
And that's it. Now we can rest assured the Xen hosts won't clobber each other's guest file systems.
Look for a follow up soon on live migration.
