Last Friday I attempted to move some of our older virtual machines hosted on do-it-yourself jailtime based Xen2 hosts. The end result is that I decided it's more trouble than it's worth, primarily because Redhat and CentOS handle guest machines in a fundamentally different way than I handled them with xen2, before there was any official vendor support.
The difference primarily lies in the partition table and boot process of the guest virtual machine. In Redhat enterprise Linux 5 (And CentOS5), the system assumes you'll be using a full partition table inside the guest machines block device. These devices are then mapped to /dev/xvd??, where as in xen2 I would map unique block devices into partitions like /dev/sda1 and /dev/sda2. The xvda devices alone aren't a big problem. The show stopping problems I ran into stem from the initial ram disk now used by the modular Xen 3 kernel, the change in hardware architecture between the old machines, and a few other factors.
Redhat also assumes you'll be using py-grub to boot the kernel and initial ram disk from inside the guest file system, rather than the host file system. This more closely resembles the behavior of a "real" physical machine, so I understand why they're doing things this way. Py-grub and the partition table ends up being a bit awkward if you want to mount the guest file system inside the host, however, as you need to use byte offset options and loopback block devices since the guest partition table isn't visible to the host.
For the indexing bots and the curious, my full attempt is logged in my wiki at ReferenceXen2toXen3onCentOS5.
Since there's now official vendor support for Xen virtual machines, I figure now's the time to switch to their way of doing things in order to prevent migration issues like this after future updates to RHEL and CentOS.
