VirtWithKVM1
KVM installation and configuration on dom04
Short description of the software and hardware
The KVM host will be named dom04.wn. The operating system will be Scientific Linux 6.1 x86_64. There are 9 ports :
- iLO3 port (IP 192.168.9.224) -> use this address in your browser to access the iLO3 Web pages of the machine;
- eth0 -> eth3 : bnx2 NIC
- eth4 -> eth7 : igb NIC
Since the rule is to always take the first available NIC as the public one, and since we will use the bnx2 NICs, then eth0 will be the public NIC and eth1 will be the private one :
eth0 --> public, no IP address; eth1 --> private, IP 192.168.10.37
The machine is an HP ProLiant DL360 G7 with 2 CPUs, each CPU having 6 hyperthreaded cores (12 cores from the OS point of view) and 72 GB of memory. There are four RAID-1 systems of 500GB each.
OS installation
- Installation was done through the iLO Integrated Remote Console
- Installation media : SL61 DVD iso's, mounted on the machine via a Virtual DVD Drive (in the Remote Console, menu "Virtual Drives" > sub-menu "CD/DVD" > select option "Virtual Image" > select the *.iso file)
- Disk space layout : one volume group using the first RAID-1 systems (sda); we will use sdb, sdc and sdd as physical disks for virtual machines. (See next section for more details)
Disk space layout in details
During the SL installation process, the wizard will ask you to add the RAID systems on which you want to install the OS. If you add all the 4 RAIDs, then the installation process will create a volume group that will spread accross the 4 RAID systems, and that's not what we want ! The volume group for the OS should be contained in the first RAID (physical device /dev/sda), and the other three RAIDs should exists as physical devices (/dev/sdb, /dev/sdc, and /dev/sdd) outside the volume group. This way, the three last RAID systems can be used as physical disks for the virtual machines.
Now speaking about the default partioning layout, if you let the installation process do the job for you, it will create three logical volumes : swap (virtual memory), home (mounted to /home) and root (mounted to /). The first device (sda) will be splitted in two physical partitions : sda1 mounted to /boot is outside the volume group, and sda2 is added to the volume group.
After the installation process is finished, you may want to correct the disk space layout. You can easily do it with the GUI called system-config-lvm :
yum install system-config-lvm system-config-lvm
(since it is a graphical, don't forget to open you ssh session on dom04 with the option -XC).
The following screenshot shows what you should get :
Image(system-config-lvm1.png,50%)
In the picture above, you can see that there is no /home : it has been deleted because not useful, and the resulting unallocated space has been added to the root logical volume. If you do so, don't forget to remove the line corresponding to /home in /etc/fstab, otherwise, your machine won't be able to reboot (during the reboot, it will try to fscheck /home, resulting in an error, and you'll be obliged to restart the machine in rescue mode to correct the /etc/fstab) !
Network configuration
Here are the NICs as they are stated in iLO :
iLO3 | iLO Dedicated Network Port | 9c:8e:99:2e:c6:a6 |
---|---|---|
iSCSI | Port 1 | 9c:8e:99:2d:e1:e5 |
iSCSI | Port 2 | 9c:8e:99:2d:e1:e7 |
iSCSI | Port 3 | 9c:8e:99:2d:e1:dd |
iSCSI | Port 4 | 9c:8e:99:2d:e1:df |
NIC | Port 1 | 9c:8e:99:2d:e1:e4 |
NIC | Port 2 | 9c:8e:99:2d:e1:e6 |
NIC | Port 3 | 9c:8e:99:2d:e1:dc |
NIC | Port 4 | 9c:8e:99:2d:e1:de |
In fact, in the table above, only the four bnx2 ports are visible (each port has two MAC addresses). There is still a PCI extension card providing four igb ports not visible in the iLO web pages. However, if you look at the output of dmesg, you should see blocks of lines like these :
igb 0000:06:00.1: irq 70 for MSI/MSI-X igb 0000:06:00.1: Intel(R) Gigabit Ethernet Network Connection igb 0000:06:00.1: eth5: (PCIe:5.0Gb/s:Width x4) f4:ce:46:a6:e5:59 igb 0000:06:00.1: eth5: PBA No: E84069-007 igb 0000:06:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Notice the different MAC address range used for igb interfaces.
eth4 | f4:ce:46:a6:e5:58 |
---|---|
eth5 | f4:ce:46:a6:e5:59 |
eth6 | f4:ce:46:a6:e5:5a |
eth7 | f4:ce:46:a6:e5:5b |
Two bridges must be created : one for the public network and another for the private network.
Here is the content of the needed network scripts in /etc/sysconfig/network-scripts/ :
# cat ifcfg-eth0 ---------------- DEVICE="eth0" BOOTPROTO="static" HWADDR="9C:8E:99:2D:E1:E4" NM_CONTROLLED="no" ONBOOT="yes" BRIDGE="br0" ---------------- # cat ifcfg-br0 ---------------- DEVICE="br0" TYPE="Bridge" NM_CONTROLLED="no" BOOTPROTO="static" ONBOOT="yes" ---------------- # cat ifcfg-eth1 ---------------- DEVICE="eth1" HWADDR="9C:8E:99:2D:E1:E6" NM_CONTROLLED="no" ONBOOT="yes" BOOTPROTO="static" BRIDGE="br1" ---------------- # cat ifcfg-br1 ---------------- DEVICE="br1" TYPE="Bridge" BOOTPROTO="static" ONBOOT="yes" IPADDR="192.168.10.37" NETMASK="255.255.0.0" ----------------
After editing these files, restart the network :
service network restart
Now, let us control that everything is ok :
# brctl show ---------------- bridge name bridge id STP enabled interfaces br0 8000.9c8e992de1e4 no eth0 br1 8000.9c8e992de1e6 no eth1 ---------------- # ifconfig ---------------- br0 Link encap:Ethernet HWaddr 9C:8E:99:2D:E1:E4 inet6 addr: fe80::9e8e:99ff:fe2d:e1e4/64 Scope:Link ... br1 Link encap:Ethernet HWaddr 9C:8E:99:2D:E1:E6 inet addr:192.168.10.37 Bcast:192.168.255.255 Mask:255.255.0.0 inet6 addr: fe80::9e8e:99ff:fe2d:e1e6/64 Scope:Link ... eth0 Link encap:Ethernet HWaddr 9C:8E:99:2D:E1:E4 inet6 addr: fe80::9e8e:99ff:fe2d:e1e4/64 Scope:Link ... eth1 Link encap:Ethernet HWaddr 9C:8E:99:2D:E1:E6 inet6 addr: fe80::9e8e:99ff:fe2d:e1e6/64 Scope:Link ... ----------------
Now, iptables must be configured to allow forwarding across the bridges :
# iptables -I FORWARD -m physdev --physdev-is-bridged -j ACCEPT # service iptables save # service iptables restart
SElinux must be disabled, otherwise you'll get strange error messages when trying to start your virtual machines :
/usr/sbin/setenforce 0 vim /etc/selinux/config --> you must set the following : SELINUX=disabled
We still have to add a default route through ccq :
route add default gw 192.168.10.100
and to make this default route permanent, don't forget to add the following line in /etc/sysconfig/network :
GATEWAY=192.168.10.100
As a result, your root table should look like this :
[root@dom05 ~]# route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 192.168.122.0 * 255.255.255.0 U 0 0 0 virbr0 link-local * 255.255.0.0 U 1015 0 0 br0 link-local * 255.255.0.0 U 1016 0 0 br1 192.168.0.0 * 255.255.0.0 U 0 0 0 br1 default ccq.wn.iihe.ac. 0.0.0.0 UG 0 0 0 br1
And then configure the DNS resolution by adding the following in /etc/resolv.conf :
search wn.iihe.ac.be nameserver 192.168.10.100 nameserver 193.190.247.140
We can now make the machine ready for SSH. Simply run :
ssh-keygen -t rsa
for key pair generation. It is also a good idea to add in .ssh/authorized_keys the public key of ccq, dom02, and dom03.
Don't forget to also add dom04.wn.iihe.ac.be in the private DNS on ccq.
RPMs installation
We simply did a :
yum groupinstall "Virtualization*"
This will install four software groups :
- Virtualization
- Virtualization Client
- Virtualization Platform
- Virtualization Tools
The following is also necessary for X forwarding with SSH :
yum install xorg-x11-xauth
After that, you must log off, and then log in back with the option -X (this log off and in is necessary to create the .Xauthority file).
Configuration of daemons
Start libvirtd :
service libvirtd start
Doing a "brctl show" or and an "ifconfig", you should see new interface virbr0.
Manage the storage pools
Launch virt-manager, right click on the virtual host (localhost), select the "Details" option in the menu, and the select the "Storage" tab in the new window. Now, we will add the three RAID systems (sdb, sdc, and sdd) to the storage pools. I will show in details the steps for the device sdb :
Image(system-config-lvm2.png,25%)
Click on the "+" button at the lower left corner of the previous window, this will launch a wizard. In the first window of the wizard, give a name to the new pool (for example "sdb"). On the next windows, fill in the fields like this :
Image(system-config-lvm3.png,25%)
Example of a VM installation from CD/DVD ISO
It should be the same as here
Example of a VM installation from PXE
It should be the same as here