DealingWithiPXE

Problem

New hypervisors come with iPXE installed instead of gPXE. This is not a problem in itself since iPXE brings a lot of new interesting features compared to gPXE. But the virtio iPXE ROM that's provided by default doesn't behave as it should with VMs that have several NICs : it only tries to boot from the first found NIC and then it stops if the boot failed, instead of trying the other NICs. For us, it's a big problem since many VMs have two NICs : the first one is the public, and the second one is the private, and that's the one that will be used to pxe-boot.

Solution

Instead of downgrading to gPXE (the past pathetic workaround), there's a better way that consists in customizing the rom by embedding an iPXE script in it. The generic method to embed scripts is described here, and iPXE scripting is explained here. We needed a script that tries all the NICs, and we have found this cool iPXE menu on GitHubGist (it does a bit more than just trying all the NICs in turn, its default behaviour...).

Here is the detailed recipe of the solution that we applied on our hypervisors :

1. Git clone the iPXE code :

git clone git://git.ipxe.org/ipxe.git

2. Get into the ipxe git dir and download the script in it :

cd ipxe
wget ...

3. Build iPXE :

cd src && make EMBED=../nic-menu.ipxe && cd ..

Note : During our first attempt, we've got an error saying that lzma.h was missing. It was fixed by installing the package xz-devel.

4. Replace the old rom by the new one :

4.1 The new virtio rom with the embedded script is ipxe/src/bin/1af41000.rom

4.2 Copy this file to the hypervisor :

scp ipxe/src/bin/1af41000.rom hypervisor:/usr/share/qemu-kvm/1af41000_new.rom

4.3 Change a symlink so that the new rom is used (on the hypervisor) :

cd /usr/share/qemu-kvm
rm pxe-virtio.rom
ln -s 1af41000_new.rom pxe-virtio.rom

If you want the details of what the new rom will do, here is the code of the embedded script :

#!ipxe

set timeout 1000

:menu
menu Network boot options for ${uuid}
item --key a default Try to boot (a)ll network adapters in turn
item
item --gap -- --- Detected network adapters ---
set i:int8 0
:loop
ifopen net${i} && item --key ${i} net${i} net(${i}): ${netX/mac} - ${netX/bustype} ${netX/busloc:busdevfn} ${pci/${netX/busloc}.0.2}:${pci/${netX/busloc}.2.2} ${netX/chip} ; ifclose
inc i
iseq ${i} 10 || goto loop
item
item --gap -- --- Alternatives ---
item --key c config Open (c)onfiguration
item --key r reboot (R)eboot computer
item --key s shell Drop to iPXE (s)hell
item --key x exit  E(x)it and continue BIOS boot order
choose --timeout ${timeout} selected && goto select || goto default
goto menu

:select
isset ${${selected}/mac} && goto nic || goto label

:nic
autoboot ${selected} && goto exit ||
echo Booting '${selected}' failed, exiting iPXE...
goto exit

:label
goto ${selected} || 
echo The label '${selected}' could not be found, returning to menu...
sleep 2
goto restart

:default
autoboot && goto exit ||
echo Booting failed, exiting iPXE...
goto exit

:config
config
goto restart

:shell
shell
goto restart

:restart
set timeout 0
goto menu

:reboot
reboot

:exit
echo Continuing BIOS boot order...
sleep 1
exit