Xen

https://wiki.debian.org/Xen

First install the hypervisor, xen aware kernel and xen tools. This can be done by a metapackage:

> apt-get install xen-linux-system

Debian Squeeze uses Grub 2 whose default is to list normal kernels first, and only then list the Xen hypervisor and its kernels.

You can change this to cause Grub to prefer to boot Xen by changing the priority of Grub's Xen configuration script (20_linux_xen) to be higher than the standard Linux config (10_linux). This is most easily done using dpkg-divert:

> dpkg-divert --divert /etc/grub.d/08_linux_xen --rename /etc/grub.d/20_linux_xen

to undo this:

> dpkg-divert --rename --remove /etc/grub.d/20_linux_xen

After any update to the Grub configuration you must apply the configuration by running:

update-grub

In order to give network access to guest domains it is necessary to configure the domain 0 network appropriately. The most common configuration is to use a software bridge.

It is recommended that you manage your own network bridge using the Debian network bridge. The Xen wiki page Host Configuration/Networking also has some useful information. The Xen supplied network scripts are not always reliable and will be removed from a later version. They are disabled by default in Debian's packages.

If you have a router that assigns ip addresses through dhcp, the following is a working example of the /etc/network/interfaces file using bridge-utils software.

> nano /etc/network/interfaces

#The loopback network interface
auto lo
iface lo inet loopback

iface eth0 inet manual

auto xenbr0
iface xenbr0 inet dhcp
   bridge_ports eth0

#other possibly useful options in a virtualized environment
  #bridge_stp off       # disable Spanning Tree Protocol
  #bridge_waitport 0    # no delay before a port becomes available
  #bridge_fd 0          # no forwarding delay

activate your changes like this:

> ifdown eth0
> killall dhclient
> ifup xenbr0
> brctl show

bridge name bridge id       STP enabled interfaces
br-lan      8000.xxxxxxxxxxxx   no      eth0

You should see your new IP address on ifconfig br-lan, and you should still be able to ping out (e.g. ping 8.8.8.8 / and resolve: ping google.com).

On a system which is dedicated to running Xen guests it is better to instead give dom0 some static amount of RAM and to disable ballooning, here 1024MB.

In order to do this you must first add the dom0_mem option to your hypervisor command line. This is done by editing /etc/default/grub and adding:

In order to do this you must first add the dom0_mem option to your hypervisor command line. This is done by editing /etc/default/grub and adding

# Xen boot parameters for all Xen boots
GRUB_CMDLINE_XEN="dom0_mem=1280M"

at the bottom of the file.

Remember to apply the change to the grub configuration by running

> update-grub

Then edit /etc/xen/xend-config.sxp to configure the toolstack to match by changing the following settings:

(dom0-min-mem 1280)
(enable-dom0-ballooning no)

At this point you should reboot so that these changes take effect.

By default, when Xen dom0 shuts down or reboots, it tries to save (i.e. hibernate) the state of the domUs. Sometimes there are problems with that - it could fail because of a lack of disk space in /var, or because of random software bugs. Because it is also clean to just have the VMs shutdown upon host shutdown, if you want you can make sure they get shut down normally by setting these parameters in /etc/default/xendomains:

XENDOMAINS_RESTORE=false
XENDOMAINS_SAVE=""

You may also want to pass some boot parameters to Xen when starting up in normal or recovery mode. Add these variables to /etc/default/grub to achieve this:

# Xen boot parameters for all Xen boots
GRUB_CMDLINE_XEN="something"
# Xen boot parameters for non-recovery Xen boots (in addition to GRUB_CMDLINE_XEN)
GRUB_CMDLINE_XEN_DEFAULT="something else"

Remember to apply the change to the grub configuration by running update-grub!

More information on the available hypervisor command line options can be found in the upstream documentation.

This information is incomplete for Squeeze and needs to be updated for Wheezy

To enable PCI pass-through, you need to know the BDF (Bus, Device, Function) id of the device. This is obtained through the lspci command, with the output containing the BDF in the format: (BB:DD.F) at the start of the line. To hide a device from Dom0 you will need to pass these boot parameters to Xen when starting. For example if using a Dom0 with 512M of memory and two devices at 01:08.1 and 01:09.2, add these variables to /etc/default/grub to achieve this:

# Xen boot parameters for all Xen boots
GRUB_CMDLINE_XEN="dom0_mem=512M pciback.hide=(01:08.1)(01:09.2)"
# Xen boot parameters for non-recovery Xen boots (in addition to GRUB_CMDLINE_XEN)
GRUB_CMDLINE_XEN_DEFAULT="something else"

For Squeeze use “pciback.hide” (kernels < 2.6.32.10), for Wheezy (I have not tested this yet) use “xen-pciback.hide”

for Squeeze you need to pass all of the devices on the bus, eg to pass any device on the 01:DD.F bus, you have pass all of them: (01:08.1)(01:09.2)(01:09.3)etc.

Remember to apply the change to the grub configuration by running

> update-grub

At least in Wheezy (not tested in Squeeze) the xen-pciback module needs to be configured through modprobe.conf and added to the initramfs additionally.

Configure the xen-pciback module by adding a modprobe include file (e.g. /etc/modprobe.d/xen-pciback.conf) with the following content (given that the PCI device would be assigned to module e1000e normally):

install e1000e /sbin/modprobe xen-pciback; /sbin/modprobe --first-time --ignore-install e1000e
options xen-pciback hide=(0000:03:00.0) 

Add the xen-pciback module to initramfs by adding it to /etc/initramfs/modules and running

> update-initramfs -u afterwards.

Please note that pci-passthrough is broken when msi is enabled (default) in Linux kernels < 3.14. Use Linux kernel >= 3.14 in DomU/VM or set pci=nomsi for DomU/VM kernel as workaround. See the following thread for detailed information: http://thread.gmane.org/gmane.comp.emulators.xen.user/81944/focus=191437

To get output from GRUB, the Xen hypervisor, the kernel and getty (login prompt) via both VGA and serial console to work, here's an example of the right settings on squeeze:

Edit /etc/default/grub and add:

GRUB_SERIAL_COMMAND="serial --unit=0 --speed=9600 --word=8 --parity=no --stop=1"
GRUB_TERMINAL="console serial"
GRUB_TIMEOUT=5
GRUB_CMDLINE_XEN="com1=9600,8n1 console=com1,vga"
GRUB_CMDLINE_LINUX="console=tty0 console=hvc0"

Here's what I used to configure the serial console (for a Supermicro X8STi-F motherboard with IPMI and SOL):

GRUB_CMDLINE_XEN="loglvl=all guest_loglvl=all com1=115200,8n1,0x3e8,5 console=com1,vga"
GRUB_CMDLINE_LINUX="console=hvc0 earlyprintk=xen"

In /etc/inittab you need at least these lines:

1:2345:respawn:/sbin/getty 38400 hvc0
2:23:respawn:/sbin/getty 38400 tty1
# NO getty on ttyS0!

This way, tty1 will show up at the VGA output, and the hvc0 will show up at the serial console.

To keep both Xen and dom0 kernel output on the same tty, just omit the “vga”-related settings from the above setup.

If you need to debug Xen and see a crash dump of the kernel, you can do it using IPMITool if your server has SOL:

ipmitool -I lanplus -H server-ip-address -U your-username sol activate | tee my-log-file.txt