Xen

https://wiki.debian.org/Xen

First install the hypervisor, xen aware kernel and xen tools. This can be done by a metapackage:

> apt-get install xen-linux-system

Debian Squeeze uses Grub 2 whose default is to list normal kernels first, and only then list the Xen hypervisor and its kernels.

You can change this to cause Grub to prefer to boot Xen by changing the priority of Grub's Xen configuration script (20_linux_xen) to be higher than the standard Linux config (10_linux). This is most easily done using dpkg-divert:

> dpkg-divert --divert /etc/grub.d/08_linux_xen --rename /etc/grub.d/20_linux_xen

to undo this:

> dpkg-divert --rename --remove /etc/grub.d/20_linux_xen

After any update to the Grub configuration you must apply the configuration by running:

update-grub

Please refer to the Network configuration section.

To avoid random, unexplained performance issues and failures in a XEN virtualization environment the following configuration items should always be checked; Correspondingly, before troubleshooting ANY XEN Host or VM issue verify the XEN Host configuration as follows:

• “Lock down” the amount of memory assigned to dom0.
• Set the amount of memory Dom0 is given at boot time
• Disable ballooning
• Do not restrict dom0 to a single VCPU
• Only use Logical Volume Manager (LVM) on Dom0 or DomU, but not both

More (for SUSE): http://www.novell.com/support/kb/doc.php?id=3559698

On a system which is dedicated to running Xen guests it is better to instead give dom0 some static amount of RAM and to disable ballooning, here 1024MB.

In order to do this you must first add the dom0_mem option to your hypervisor command line. This is done by editing /etc/default/grub and adding:

In order to do this you must first add the dom0_mem option to your hypervisor command line. This is done by editing /etc/default/grub and adding

# Xen boot parameters for all Xen boots
GRUB_CMDLINE_XEN="dom0_mem=1280M"

at the bottom of the file.

Remember to apply the change to the grub configuration by running

> update-grub

Then edit /etc/xen/xend-config.sxp to configure the toolstack to match by changing the following settings:

(dom0-min-mem 1280)
(enable-dom0-ballooning no)

At this point you should reboot so that these changes take effect.

> cat /proc/xen/capabilities

Should display “control_d”

By default, when Xen dom0 shuts down or reboots, it tries to save (i.e. hibernate) the state of the domUs. Sometimes there are problems with that - it could fail because of a lack of disk space in /var, or because of random software bugs. Because it is also clean to just have the VMs shutdown upon host shutdown, if you want you can make sure they get shut down normally by setting these parameters in /etc/default/xendomains:

XENDOMAINS_RESTORE=false
XENDOMAINS_SAVE=""

You may also want to pass some boot parameters to Xen when starting up in normal or recovery mode. Add these variables to /etc/default/grub to achieve this:

# Xen boot parameters for all Xen boots
GRUB_CMDLINE_XEN="something"
# Xen boot parameters for non-recovery Xen boots (in addition to GRUB_CMDLINE_XEN)
GRUB_CMDLINE_XEN_DEFAULT="something else"

Remember to apply the change to the grub configuration by running update-grub!

More information on the available hypervisor command line options can be found in the upstream documentation.

This information is incomplete for Squeeze and needs to be updated for Wheezy

To enable PCI pass-through, you need to know the BDF (Bus, Device, Function) id of the device. This is obtained through the lspci command, with the output containing the BDF in the format: (BB:DD.F) at the start of the line. To hide a device from Dom0 you will need to pass these boot parameters to Xen when starting. For example if using a Dom0 with 512M of memory and two devices at 01:08.1 and 01:09.2, add these variables to /etc/default/grub to achieve this:

# Xen boot parameters for all Xen boots
GRUB_CMDLINE_XEN="dom0_mem=512M pciback.hide=(01:08.1)(01:09.2)"
# Xen boot parameters for non-recovery Xen boots (in addition to GRUB_CMDLINE_XEN)
GRUB_CMDLINE_XEN_DEFAULT="something else"

For Squeeze use “pciback.hide” (kernels < 2.6.32.10), for Wheezy (I have not tested this yet) use “xen-pciback.hide”

for Squeeze you need to pass all of the devices on the bus, eg to pass any device on the 01:DD.F bus, you have pass all of them: (01:08.1)(01:09.2)(01:09.3)etc.

Remember to apply the change to the grub configuration by running

> update-grub

At least in Wheezy (not tested in Squeeze) the xen-pciback module needs to be configured through modprobe.conf and added to the initramfs additionally.

Configure the xen-pciback module by adding a modprobe include file (e.g. /etc/modprobe.d/xen-pciback.conf) with the following content (given that the PCI device would be assigned to module e1000e normally):

install e1000e /sbin/modprobe xen-pciback; /sbin/modprobe --first-time --ignore-install e1000e
options xen-pciback hide=(0000:03:00.0) 

Add the xen-pciback module to initramfs by adding it to /etc/initramfs/modules and running

> update-initramfs -u afterwards.

Please note that pci-passthrough is broken when msi is enabled (default) in Linux kernels < 3.14. Use Linux kernel >= 3.14 in DomU/VM or set pci=nomsi for DomU/VM kernel as workaround. See the following thread for detailed information: http://thread.gmane.org/gmane.comp.emulators.xen.user/81944/focus=191437

To get output from GRUB, the Xen hypervisor, the kernel and getty (login prompt) via both VGA and serial console to work, here's an example of the right settings on squeeze:

Edit /etc/default/grub and add:

GRUB_SERIAL_COMMAND="serial --unit=0 --speed=9600 --word=8 --parity=no --stop=1"
GRUB_TERMINAL="console serial"
GRUB_TIMEOUT=5
GRUB_CMDLINE_XEN="com1=9600,8n1 console=com1,vga"
GRUB_CMDLINE_LINUX="console=tty0 console=hvc0"

Here's what I used to configure the serial console (for a Supermicro X8STi-F motherboard with IPMI and SOL):

GRUB_CMDLINE_XEN="loglvl=all guest_loglvl=all com1=115200,8n1,0x3e8,5 console=com1,vga"
GRUB_CMDLINE_LINUX="console=hvc0 earlyprintk=xen"

In /etc/inittab you need at least these lines:

1:2345:respawn:/sbin/getty 38400 hvc0
2:23:respawn:/sbin/getty 38400 tty1
# NO getty on ttyS0!

This way, tty1 will show up at the VGA output, and the hvc0 will show up at the serial console.

To keep both Xen and dom0 kernel output on the same tty, just omit the “vga”-related settings from the above setup.

If you need to debug Xen and see a crash dump of the kernel, you can do it using IPMITool if your server has SOL:

ipmitool -I lanplus -H server-ip-address -U your-username sol activate | tee my-log-file.txt