{"id":498,"date":"2011-06-23T19:57:04","date_gmt":"2011-06-23T19:57:04","guid":{"rendered":"http:\/\/www.redwireservices.com\/?p=498"},"modified":"2014-07-25T16:16:43","modified_gmt":"2014-07-25T23:16:43","slug":"application-availability-migrate-physical-linux-server-to-vmware-p2v","status":"publish","type":"post","link":"https:\/\/www.redwireservices.com\/application-availability-migrate-physical-linux-server-to-vmware-p2v","title":{"rendered":"Application Availability: Migrate Physical Linux Server to VMware (P2V)"},"content":{"rendered":"

As part of my work helping companies better protect their data and applications, we also look at the\u00a0availability\u00a0of their applications. \u00a0One great way to improve\u00a0availability\u00a0of applications to is move them to a virtualization platform, such as VMware Virtual Infrastructure<\/a>.<\/p>\n

VMware, and other virtualization platforms, provide a great deal of\u00a0flexibility\u00a0to drastically minimize downtime of your critical applications. \u00a0This is accomplished by abstracting out the application from a physical\u00a0piece\u00a0of hardware and placing this abstracted “block” (Virtual Machine) onto a “private cloud” of \u00a0hardware. \u00a0In this configuration if a\u00a0piece\u00a0of hardware fails, or needs to be taken offline for maintenance, the application Virtual Machine (VM) can simply be moved to another physical server. \u00a0If you have the means to deploy their enterprise solutions, this can be\u00a0achieved\u00a0automatically when hardware problems occur without any human intervention, even when working with legacy applications that are not “cluster aware.”<\/p>\n

That’s great, so how do we move our physical servers over to VMware? \u00a0Well, for Windows there are built in plugins and standalone applications to do this (search for VMware Converter\/VMware Converter Standalone). \u00a0It appears that recently they’ve added some Linux support, but I’ve constructed my own method which has one large advantage: \u00a0being able to “pre seed” large data sets before switch-over. \u00a0Put another way, if you have a lot of data on the server you wish to move, this method will allow you to copy the data slowly over days or weeks while the server is still online and available. \u00a0Then when the data is fully\u00a0replicated, you can quickly cut over to the new server — usually in less than an hour.<\/p>\n

This is a heavy task and should not be taken\u00a0lightly. \u00a0Make sure your Linux skills are sharp, and that you have great backups in place should something go wrong. \u00a0You’ve been warned<\/strong>…<\/p>\n

Let’s walk through the process. \u00a0At a high level, here is what we need to do:<\/p>\n

    \n
  1. Create the target VM on VMware<\/li>\n
  2. Seed data<\/li>\n
  3. Configure boot code — this can be tricky!<\/li>\n
  4. Cutover<\/li>\n<\/ol>\n

    Let’s take a look at each of these steps in a little more detail.<\/span><\/span><\/p>\n

    Create Target VM<\/h2>\n
      \n
    1. Boot this VM from a rescue CD provided by your distribution of choice (I prefer\u00a0System Rescue CD<\/a>)<\/li>\n
    2. Create the needed partitions, file systems, and LVM volumes to match the source system. \u00a0Note: \u00a0the new file systems need not be the same size as the source, they just need enough space for the actual data to be moved.<\/li>\n<\/ol>\n

       <\/p>\n

      Start off by inspecting the existing configuration on the source server. \u00a0Look at the output of mount<\/em>, fdisk -l, pvs, vgs, lvs, <\/em>and the content of \/etc\/fstab.<\/em><\/span><\/span><\/p>\n

      For \u00a0example, take this fstab <\/em>from a server I recently converted (Ubunut 8.04, with the interesting lines in bold<\/strong>):<\/span><\/p>\n

      # \/etc\/fstab: static file system information. # # <file system> <mount point> <type>\u00a0 <options>\u00a0<dump> <pass> proc\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \/proc\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 proc\u00a0\u00a0\u00a0 defaults 0 0 \/dev\/system\/root \/\u00a0\u00a0ext3\u00a0\u00a0\u00a0 relatime,errors=remount-ro 0 1<\/strong> \/dev\/sda1 \/boot\u00a0\u00a0ext3\u00a0\u00a0\u00a0 relatime\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 2<\/strong> \/dev\/system\/home \/home\u00a0ext3\u00a0relatime\u00a0\u00a0\u00a00 2<\/strong> \/dev\/system\/tmp \/tmp\u00a0ext3\u00a0relatime\u00a0\u00a00\u00a0 2<\/strong> \/dev\/system\/usr \/usr\u00a0\u00a0ext3\u00a0relatime\u00a00\u00a02<\/strong> \/dev\/system\/var \/var\u00a0ext3\u00a0\u00a0\u00a0 relatime\u00a00\u00a02<\/strong> \/dev\/system\/opt \/opt ext3\u00a0\u00a0\u00a0 relatime\u00a00\u00a02<\/strong> \/dev\/system\/swap none\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 swap\u00a0\u00a0\u00a0 sw 0 0<\/strong> \/dev\/scd0\u00a0\u00a0\/media\/cdrom0 udf,iso9660 user,noauto,exec,utf8 0 0<\/div>\n

      On the new VM, I need to mirror these filesystems. \u00a0I could choose to consolidate them into one large root volume, but I’ll just stick with what is there. \u00a0If you choose to change the layout, make sure to save your new version of \/etc\/fstab on the new server and protect it with an –exclude=\/etc\/fstab <\/em>on your rsync<\/em> command for copying data. \u00a0To\u00a0replicate\u00a0the current setup, I:<\/span><\/span><\/p>\n

      Setup the needed physical partitions using fdisk, creating a boot partition (\/dev\/sda1) and a LVM physical volume (\/dev\/sda2):<\/span><\/p>\n

      sudo fdisk \/dev\/sda<\/span><\/pre>\n
      Next I create the needed LVM physical volume, group, and logical volumes:<\/span><\/p>\n
      pvcreate \/dev\/sda2 <\/span>vgcreate system<\/span>\/dev\/sda2 <\/span>lvcreate -n root -L500M \/dev\/system <\/span>lvcreate -n home -L2G \/dev\/system <\/span>lvcreate -n tmp -L2G \/dev\/system <\/span>lvcreate -n usr -L4G \/dev\/system <\/span>lvcreate -n var -L4G \/dev\/system <\/span>lvcreate -n opt -L4G \/dev\/system <\/span>lvcreate -n swap -L4G \/dev\/system<\/span><\/pre>\n
      Next create the filesystems and swap partition:<\/span><\/p>\n
      mkswap \/dev\/system\/swap mkfs.ext3 \/dev\/sda1 for fs in root home tmp usr var opt\u00a0 do \u00a0 \u00a0mkfs.ext3 \/dev\/system\/${fs} \u00a0 \u00a0echo DONE creating ${fs} done <\/span><\/pre>\n
      Now that the filesystem structure exists, we can create mount points and mount them up:<\/span><\/p>\n
      mkdir \/mnt\/new-system <\/span># mount the root volume first! mount \/dev\/system\/root \/mnt\/new-system <\/span> <\/span> cd \/mnt\/new-system <\/span><\/span>for fs in home tmp usr var opt\r\ndo\r\n\u00a0 \u00a0mkdir \/mnt\/new-system\/${fs}\r\n   mount \/dev\/system\/${fs} \/mnt\/new-system\/${fs}\r\ndone\u00a0<\/span><\/pre>\n
      Seed Data<\/h2>\n
      Now that we have the new system-root in place at \/mnt\/new-system, we can do our\u00a0initial\u00a0copy using <\/span><\/span>rsync<\/span><\/em> over ssh:<\/em><\/span><\/p>\n
      rsync --progress --exclude=\/proc --exclude=\/sys --exclude=\/etc\/fstab --bwlimit=4096 -ave ssh root@old-server:\/ \/mnt\/new-system <\/em><\/span><\/pre>\n
      The above command will replicate everything from the root of the old server, to \/mnt\/new-system in our newly created VM. \u00a0You may add additional <\/span>–exclude<\/em> statements to exclude files that are no longer needed. \u00a0Adjust the <\/span>–bwlimit<\/em> argument to the number of Kbps you wish to copy, in this case we limit the amount of bandwidth to 4Mbps as to not\u00a0interfere\u00a0with production operations.<\/span><\/span><\/p>\n
      Configure the Bootloader<\/h2>\n
      Great, now we have a new VM with all file systems copied over. \u00a0The next step is the trickiest part to get right: \u00a0installing GRUB (or other) bootloader. \u00a0The bootloader is the program used to start your computer right after the BIOS<\/a> and it needs to know just enough about our VM to load our kernel and get going.<\/span><\/span><\/p>\n
      Below is what has worked for me, but in my\u00a0experience\u00a0every conversion I’ve done has required some last minute changes to make things work.<\/span><\/span><\/span><\/p>\n
      On Ubuntu (tested lately on 8.04); l<\/span><\/span><\/span>oad the\u00a0mptscsih module in your recovery environment (so that GRUB can see your new VMware disks):<\/span><\/p>\n
      modprobe\u00a0mptscsih<\/span><\/span><\/span><\/pre>\n
      Next change the current root to be our new \/mnt\/new-system directory and build a new initial ram disk with the proper drivers for our new system:<\/span><\/span><\/span><\/p>\n
      chroot \/mnt\/new-server \/bin\/bash # Use your current kernel version here (find it with uname -a<\/em> on the source system) update-initramfs -u -k '2.6.24-16-server<\/em>' <\/span><\/span><\/span><\/pre>\n
      Next hit CTRL+D<\/em> to exit the chroot environment. \u00a0Next we install GRUB.<\/span><\/span><\/span><\/p>\n
      grub-install --root-directory=\/mnt\/new-server \/dev\/sda <\/span><\/span><\/span><\/pre>\n
      Now inspect \/mnt\/new-server\/boot\/menu.lst (or\u00a0wherever\u00a0your GRUB config file is, could be in boot\/grub.conf or other file) and ensure that the kernel paths are correct. \u00a0You may infer the correct paths from boot\/grub\/device.map. \u00a0If the wrong path is used, update menu.lst<\/em> or grub.conf<\/em> with the correct entry (such as hd(0,0)) <\/em>and re-run grub-install<\/em>.<\/span><\/span><\/span><\/p>\n
      With RedHat or CentOS (I’ve done this with RHEL4\/5 and CentOS 4\/5) the steps are essentially the same, however I’ve found it easiest here to rebuild the kernel on the source<\/strong> server with these options:<\/p>\n
      sudo mkinitrd -v -f --with=mptscsih --with=mptspi \/boot\/initrd-`uname -r`-VM<\/strong>.img `uname -r`<\/pre>\n
      Then copy this kernel ending in “-VM” to the new VM in place of the\u00a0original\u00a0file, then install GRUB.<\/p>\n
      The next step is to reboot and test that your boot code works. \u00a0Disconnect<\/strong> any network connections to avoid conflicts with the existing server, and reboot. \u00a0Upon boot note any errors you may need to correct, often times at this point I discover one of the following issues:<\/p>\n
        \n
      1. I messed up with the mptscsih module and the root volume can’t be found, kernel panic<\/li>\n
      2. There was a typo in \/etc\/fstab or a volume was missed<\/li>\n
      3. Some network dependent services fail or take forever to start — this is normal as the network is down for this VM.<\/li>\n<\/ol>\n
        If you get to a login screen, congratulations the hard part is over! \u00a0If not, fix the noted issues and try again.<\/p>\n
        Cutover<\/h2>\n
        Now that your new VM is booting it’s time to schedule a cutover time\/date. \u00a0When that time arrives:<\/p>\n
          \n
        1. Power down current production system, boot from\u00a0Rescue\u00a0CD<\/li>\n
        2. Mount existing filesystems in under \/mnt, much like we did for the rescue system<\/li>\n
        3. Boot the VM from the Rescue CD, and re-mount partitions under \/mnt\/new-system<\/li>\n
        4. Initiate final data copy, grabbing files changed since the seed operation (and previous open\/inconsistent\u00a0files such as MySQL databases)<\/li>\n
        5. Reboot, cross your fingers, and test\u00a0thoroughly<\/li>\n<\/ol>\n
          For step 2 I use this rsync <\/em>command below on the new VM (don’t use\u00a0–xattrs<\/em> if your system doesn’t have SELinux capabilities; but you must use it if that is in use):<\/p>\n
          rsync --delete<\/strong> --exclude=\/boot\u00a0--exclude=\/proc --exclude=\/etc\/fstab\u00a0--exclude=\/sys --xattrs -ave ssh \/ root@old-server:\/mnt\/new-server<\/pre>\n
          The –delete<\/strong> option removes files from the destination (in our case, the VM) that no longer exist on the physical server. \u00a0If you are afraid of messing up this command, you can add the –dry-run<\/strong> argument to see which files will be deleted.<\/p>\n
          It is very important that this command be run from the new VM and that the order of hosts is SOURCE<\/strong> then DESTINATION <\/strong>(destination is the new VM in this case), if you mess this up you could easily destroy your installation.<\/p>\n
          After this rsync<\/em> finishes, power off the old physical server and reboot the VM with the virtual network\u00a0adapters\u00a0connected.<\/p>\n
          Hopefully everything comes up and runs fine at this point, you can test by accessing the services installed on this server. \u00a0A\u00a0lot of things can go wrong at this point, however, here are some things I’ve encountered and how to deal with them.<\/p>\n
          On Ubuntu, the\u00a0Ethernet\u00a0device will change when moving to VMware, for example from eth0<\/strong> to eth1<\/strong>. \u00a0This can confuse installed applications configured to use a specific\u00a0adapter, and static IPs used on the old server will no longer be configured.<\/p>\n
          This can be corrected by editing\u00a0\/etc\/udev\/rules.d\/70-persistent-net.rules and commenting out the old rule for eth0<\/strong>, and changing eth1<\/strong> in the new line to eth0<\/strong>, here is my file post change:<\/p>\n
          # PCI device 0x8086:0x100e (e1000)\r\n#SUBSYSTEM==\"net\", ACTION==\"add\", DRIVERS==\"?*\", ATTR{address}==\"00:11:11:19:3e:df\", ATTR{type}==\"1\", KERNEL==\"eth*\", NAME=\"eth0\"\r\n\r\n# PCI device 0x1022:0x2000 (pcnet32)\r\nSUBSYSTEM==\"net\", ACTION==\"add\", DRIVERS==\"?*\", ATTR{address}==\"00:50:56:9e:00:06\", ATTR{type}==\"1\", KERNEL==\"eth*\", NAME=\"eth0\"<\/pre>\n
          I’ve also seen the loopback adapter fail after a move on Ubuntu (lo, 127.0.0.1), which many applications rely on. \u00a0I fixed this by adding back the line below in \/etc\/hosts (don’t ask me how it dissapeared):<\/p>\n
          127.0.0.1 \u00a0 \u00a0 \u00a0 localhost<\/pre>\n
          That pretty much sums it up. \u00a0If I were you, I’d keep the previous server around for a week or two, just in case some service is discovered that did not come back properly you’ll have the option to review the old server. \u00a0I’d\u00a0recommend\u00a0pulling out the network and power cables from the old server at this point as well, just to make sure someone doesn’t accidentally power it back on.<\/p>\n
          I hope you’ve found this article helpful, feel free to leave me feedback on typos or other snags you’ve ran into so we can make this article better.<\/p>\n","protected":false},"excerpt":{"rendered":"
          As part of my work helping companies better protect their data and applications, we also look at the\u00a0availability\u00a0of their applications. \u00a0One great way to improve\u00a0availability\u00a0of applications to is move them to a virtualization platform, such as VMware Virtual Infrastructure. VMware, … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[31,3,11],"tags":[8,6,9,10,5,7],"_links":{"self":[{"href":"https:\/\/www.redwireservices.com\/wp-json\/wp\/v2\/posts\/498"}],"collection":[{"href":"https:\/\/www.redwireservices.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.redwireservices.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.redwireservices.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.redwireservices.com\/wp-json\/wp\/v2\/comments?post=498"}],"version-history":[{"count":36,"href":"https:\/\/www.redwireservices.com\/wp-json\/wp\/v2\/posts\/498\/revisions"}],"predecessor-version":[{"id":1489,"href":"https:\/\/www.redwireservices.com\/wp-json\/wp\/v2\/posts\/498\/revisions\/1489"}],"wp:attachment":[{"href":"https:\/\/www.redwireservices.com\/wp-json\/wp\/v2\/media?parent=498"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.redwireservices.com\/wp-json\/wp\/v2\/categories?post=498"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.redwireservices.com\/wp-json\/wp\/v2\/tags?post=498"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}