Differences

This shows you the differences between two versions of the page.

--- hypervisor:base:xen [2014/03/24 23:31]
admin [Configure Domain 0 Memory]
+++ hypervisor:base:xen [2014/04/03 02:07] (current)
admin Links adapted because of a move operation
@@ Line 1: / Line 1: @@
 ====== Xen ======
+<WRAP center round important 60%>
+In case you want to install the full XCP-XAPI toolstack, don't go any further and read the [[xcp-xapi|XCP-XAPI]] section.
+</WRAP>
 [[https://wiki.debian.org/Xen]]
@@ Line 27: / Line 31: @@
 ===== Configure Networking =====
-In order to give network access to guest domains it is necessary to configure the domain 0 network appropriately. The most common configuration is to use a software bridge.
+Please refer to the [[:hypervisor:base:network|Network configuration]] section.
-It is recommended that you manage your own network bridge using the [[https://wiki.debian.org/BridgeNetworkConnections|Debian network bridge]]. The Xen wiki page [[http://wiki.xen.org/wiki/Host%20Configuration/Networking|Host Configuration/Networking]] also has some useful information. The Xen supplied network scripts are not always reliable and will be removed from a later version. They are disabled by default in Debian's packages.
+===== Reliable Setup =====
-If you have a router that assigns ip addresses through dhcp, the following is a working example of the **/etc/network/interfaces** file using bridge-utils software.
+To avoid random, unexplained performance issues and failures in a XEN virtualization environment the following configuration items should always be checked;  Correspondingly, before troubleshooting ANY XEN Host or VM issue verify the XEN Host configuration as follows:
+  * "Lock down" the amount of memory assigned to dom0.
+  * Set the amount of memory Dom0 is given at boot time
+  * Disable ballooning
+  * Do not restrict dom0 to a single VCPU
+  * Only use Logical Volume Manager (LVM) on Dom0 or DomU, but not both
+More (for SUSE): [[http://www.novell.com/support/kb/doc.php?id=3559698]]
+==== Configure Dom0 Memory ====
+On a system which is dedicated to running Xen guests it is better to instead give dom0 some static amount of RAM and to disable ballooning, here 1024MB.
+In order to do this you must first add the dom0_mem option to your hypervisor command line. This is done by editing /etc/default/grub and adding:
+In order to do this you must first add the dom0_mem option to your hypervisor command line. This is done by editing **/etc/default/grub** and adding
 <code>
-> nano /etc/network/interfaces
+# Xen boot parameters for all Xen boots
+GRUB_CMDLINE_XEN="dom0_mem=1280M"
+</code>
-#The loopback network interface
+at the bottom of the file.
-auto lo
-iface lo inet loopback
-iface eth0 inet manual
+<WRAP center round info 80%>
+Note : On servers with huge memory, Xen kernel crash. You must set a dom0 memory limit.
-auto xenbr0
+Take care on Wheezy, 1024M is not enough and cause kernel crash at boot with out-of-memory message.
-iface xenbr0 inet dhcp
+</WRAP>
-   bridge_ports eth0
-#other possibly useful options in a virtualized environment
+Remember to apply the change to the grub configuration by running
-  #bridge_stp off       # disable Spanning Tree Protocol
+<code>
-  #bridge_waitport 0    # no delay before a port becomes available
+> update-grub
-  #bridge_fd 0          # no forwarding delay
 </code>
-activate your changes like this:
+Then edit **/etc/xen/xend-config.sxp** to configure the toolstack to match by changing the following settings:
 <code>
-> ifdown eth0
+(dom0-min-mem 1280)
-> killall dhclient
+(enable-dom0-ballooning no)
-> ifup xenbr0
+</code>
-> brctl show
-bridge name bridge id       STP enabled interfaces
+At this point you should **reboot** so that these changes take effect.
-br-lan      8000.xxxxxxxxxxxx   no      eth0
+=== Ensure that Xen is running ===
+<code>
+> cat /proc/xen/capabilities
 </code>
+Should display “control_d”
+===== Configure guest behaviour on host reboot =====
-You should see your new IP address on ifconfig br-lan, and you should still be able to ping out (e.g. ping 8.8.8.8 / and resolve: ping google.com).
+By default, when Xen dom0 shuts down or reboots, it tries to save (i.e. hibernate) the state of the domUs. Sometimes there are problems with that - it could fail because of a lack of disk space in /var, or because of random software bugs. Because it is also clean to just have the VMs shutdown upon host shutdown, if you want you can make sure they get shut down normally by setting these parameters in **/etc/default/xendomains**:
+<code>
+XENDOMAINS_RESTORE=false
+XENDOMAINS_SAVE=""
+</code>
+===== Configure Boot Parameters =====
-===== Configure Domain 0 Memory =====
+You may also want to pass some boot parameters to Xen when starting up in normal or recovery mode. Add these variables to /etc/default/grub to achieve this:
+<code>
+# Xen boot parameters for all Xen boots
+GRUB_CMDLINE_XEN="something"
+# Xen boot parameters for non-recovery Xen boots (in addition to GRUB_CMDLINE_XEN)
+GRUB_CMDLINE_XEN_DEFAULT="something else"
+</code>
-On a system which is dedicated to running Xen guests it is better to instead give dom0 some static amount of RAM and to disable ballooning, here 1024MB.
+Remember to apply the change to the grub configuration by running update-grub!
-In order to do this you must first add the dom0_mem option to your hypervisor command line. This is done by editing /etc/default/grub and adding:
+More information on the available hypervisor command line options can be found in the [[http://xenbits.xen.org/docs/unstable/misc/xen-command-line.html|upstream documentation]]. =====
+===== Configure PCI pass-through Parameters =====
+This information is incomplete for Squeeze and needs to be updated for Wheezy
-In order to do this you must first add the dom0_mem option to your hypervisor command line. This is done by editing **/etc/default/grub*$ and adding
+To enable PCI pass-through, you need to know the BDF (Bus, Device, Function) id of the device. This is obtained through the lspci command, with the output containing the BDF in the format: (BB:DD.F) at the start of the line. To hide a device from Dom0 you will need to pass these boot parameters to Xen when starting. For example if using a Dom0 with 512M of memory and two devices at 01:08.1 and 01:09.2, add these variables to **/etc/default/grub** to achieve this:
 <code>
 # Xen boot parameters for all Xen boots
-GRUB_CMDLINE_XEN="dom0_mem=1024M"
+GRUB_CMDLINE_XEN="dom0_mem=512M pciback.hide=(01:08.1)(01:09.2)"
+# Xen boot parameters for non-recovery Xen boots (in addition to GRUB_CMDLINE_XEN)
+GRUB_CMDLINE_XEN_DEFAULT="something else"
 </code>
-at the bottom of the file.
+For Squeeze use "pciback.hide" (kernels < 2.6.32.10), for Wheezy (I have not tested this yet) use "xen-pciback.hide"
-<WRAP center round info 80%>
+for Squeeze you need to pass all of the devices on the bus, eg to pass any device on the 01:DD.F bus, you have pass all of them: (01:08.1)(01:09.2)(01:09.3)etc.
-Note : On servers with huge memory, Xen kernel crash. You must set a dom0 memory limit. Take care on Wheezy, 1024M is not enough and cause kernel crash at boot with out-of-memory message.
-</WRAP>
 Remember to apply the change to the grub configuration by running
@@ Line 88: / Line 125: @@
 </code>
-Then edit **/etc/xen/xend-config.sxp** to configure the toolstack to match by changing the following settings:
+At least in Wheezy (not tested in Squeeze) the xen-pciback module needs to be configured through //modprobe.conf// and added to the initramfs additionally.
+Configure the xen-pciback module by adding a modprobe include file (e.g. **/etc/modprobe.d/xen-pciback.conf**) with the following content (given that the PCI device would be assigned to module e1000e normally):
 <code>
-(dom0-min-mem 1024)
+install e1000e /sbin/modprobe xen-pciback; /sbin/modprobe --first-time --ignore-install e1000e
-(enable-dom0-ballooning no)
+options xen-pciback hide=(0000:03:00.0)
 </code>
-At this point you should reboot so that these changes take effect.
+Add the xen-pciback module to initramfs by adding it to **/etc/initramfs/modules** and running
+<code>
+> update-initramfs -u afterwards.
+</code>
+Please note that pci-passthrough is broken when msi is enabled (default) in Linux kernels < 3.14. Use Linux kernel >= 3.14 in DomU/VM or set pci=nomsi for DomU/VM kernel as workaround. See the following thread for detailed information: [[http://thread.gmane.org/gmane.comp.emulators.xen.user/81944/focus=191437]]
+===== Enable Serial Console =====
+To get output from GRUB, the Xen hypervisor, the kernel and getty (login prompt) via both VGA and serial console to work, here's an example of the right settings on squeeze:
+Edit **/etc/default/grub** and add:
+<code>
+GRUB_SERIAL_COMMAND="serial --unit=0 --speed=9600 --word=8 --parity=no --stop=1"
+GRUB_TERMINAL="console serial"
+GRUB_TIMEOUT=5
+GRUB_CMDLINE_XEN="com1=9600,8n1 console=com1,vga"
+GRUB_CMDLINE_LINUX="console=tty0 console=hvc0"
+</code>
+Here's what I used to configure the serial console (for a Supermicro X8STi-F motherboard with IPMI and SOL):
+<code>
+GRUB_CMDLINE_XEN="loglvl=all guest_loglvl=all com1=115200,8n1,0x3e8,5 console=com1,vga"
+GRUB_CMDLINE_LINUX="console=hvc0 earlyprintk=xen"
+</code>
+In **/etc/inittab** you need at least these lines:
+<code>
+:2345:respawn:/sbin/getty 38400 hvc0
+:23:respawn:/sbin/getty 38400 tty1
+# NO getty on ttyS0!
+</code>
+This way, tty1 will show up at the VGA output, and the hvc0 will show up at the serial console.
+To keep both Xen and dom0 kernel output on the same tty, just omit the "vga"-related settings from the above setup.
+If you need to debug Xen and see a crash dump of the kernel, you can do it using IPMITool if your server has SOL:
+<code>
+ipmitool -I lanplus -H server-ip-address -U your-username sol activate | tee my-log-file.txt
+</code>