DealingWithiPXE
Problem
New hypervisors come with iPXE installed instead of gPXE. This is not a problem in itself since iPXE brings a lot of new interesting features compared to gPXE. But the virtio iPXE ROM that's provided by default doesn't behave as it should with VMs that have several NICs : it only tries to boot from the first found NIC and then it stops if the boot failed, instead of trying the other NICs. For us, it's a big problem since many VMs have two NICs : the first one is the public, and the second one is the private, and that's the one that will be used to pxe-boot.
Solution
Instead of downgrading to gPXE (the past pathetic workaround), there's a better way that consists in customizing the rom by embedding an iPXE script in it. The generic method to embed scripts is described here, and iPXE scripting is explained here. We needed a script that tries all the NICs, and we have found this cool iPXE menu on GitHubGist (it does a bit more than just trying all the NICs in turn, its default behaviour...).
Here is the detailed recipe of the solution that we applied on our hypervisors :
1. Git clone the iPXE code :
git clone git://git.ipxe.org/ipxe.git
2. Get into the ipxe git dir and download the script in it :
cd ipxe wget ...
3. Build iPXE :
cd src && make EMBED=../nic-menu.ipxe && cd ..
Note : During our first attempt, we've got an error saying that lzma.h was missing. It was fixed by installing the package xz-devel.
4. Replace the old rom by the new one :
4.1 The new virtio rom with the embedded script is ipxe/src/bin/1af41000.rom
4.2 Copy this file to the hypervisor :
scp ipxe/src/bin/1af41000.rom hypervisor:/usr/share/qemu-kvm/1af41000_new.rom
4.3 Change a symlink so that the new rom is used (on the hypervisor) :
cd /usr/share/qemu-kvm rm pxe-virtio.rom ln -s 1af41000_new.rom pxe-virtio.rom
If you want the details of what the new rom will do, here is the code of the embedded script :
#!ipxe set timeout 1000 :menu menu Network boot options for ${uuid} item --key a default Try to boot (a)ll network adapters in turn item item --gap -- --- Detected network adapters --- set i:int8 0 :loop ifopen net${i} && item --key ${i} net${i} net(${i}): ${netX/mac} - ${netX/bustype} ${netX/busloc:busdevfn} ${pci/${netX/busloc}.0.2}:${pci/${netX/busloc}.2.2} ${netX/chip} ; ifclose inc i iseq ${i} 10 || goto loop item item --gap -- --- Alternatives --- item --key c config Open (c)onfiguration item --key r reboot (R)eboot computer item --key s shell Drop to iPXE (s)hell item --key x exit E(x)it and continue BIOS boot order choose --timeout ${timeout} selected && goto select || goto default goto menu :select isset ${${selected}/mac} && goto nic || goto label :nic autoboot ${selected} && goto exit || echo Booting '${selected}' failed, exiting iPXE... goto exit :label goto ${selected} || echo The label '${selected}' could not be found, returning to menu... sleep 2 goto restart :default autoboot && goto exit || echo Booting failed, exiting iPXE... goto exit :config config goto restart :shell shell goto restart :restart set timeout 0 goto menu :reboot reboot :exit echo Continuing BIOS boot order... sleep 1 exit