Motivation

I provide many different IT services and some of them depend on different system libraries/versions, the same database system in different versions and so on and so forth. Certainly it’s possible to resolve all this dependencies and side effects on a single host with a single OS version but it sometimes you hate yourself for such setups.

Hardware

Setup

Preparation

I presume that there’s a running minimal Ubuntu Xenial installation on an AX60-SSD with remaining disk space in volume group vg0. Maybe with full disk encryption like described in the last post.
If there is no remeining disk space (default on Hetzner hosts) just replace all LVM specific guest settings with file based images.

Installation

Install necessary software: apt-get install xen-hypervisor-4.6-amd64 xen-tools

Because Hetzner provides only one IPv4 (of course you can buy additional IPs) I use an RFC1918 compliant /16 network with NATing and a reverse proxy for HTTP/HTTPs on dom0.
The simplest approach is to create a new bridge with a dummy interface where the dummy interface is used to add a local IP for dom0.

Enable dummy module and persist it:

modprobe dummy
cat <<EOF > /etc/modprobe.d/xen.conf
alias dummy0 dummy
options dummy numdummies=1
EOF

Persist bridge and activate new config:

cat <<EOF >> /etc/network/interfaces
auto dummy0 xenbr0
iface xenbr0 inet static
  address 10.0.0.1
  netmask 255.255.0.0
  gateway 10.0.0.1
  bridge_ports dummy0

iface dummy0 inet manual
EOF
systemctl restart networking.service

Of course you can use any RFC1918 compliant IP range. I really like IPv6 but I see no advantage for this special use case an therefore all guest are IPv4 only.

Configure dom0 to NAT the private network:

cat <<EOF > /etc/sysctl.d/99-xen.conf 
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
net.ipv4.ip_forward = 1
EOF
sysctl -p /etc/sysctl.d/99-xen.conf

Configure iptables:

cat <<EOF > /etc/rc.local
# Cleanup
while iptables -t nat -D PREROUTING 1 > /dev/null 2>&1; do sleep 0; done
while iptables -t nat -D POSTROUTING 1 > /dev/null 2>&1; do sleep 0; done

# XEN NATing
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
# Example for DNAT mapping (forward public TCP port 9999 to guest on 10.0.0.99)
# iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 9999 -j DNAT --to 10.0.0.99

# XEN domU to dom0 access if other iptables rules are active
# iptables -A INPUT -p tcp --dport 5432 -s 10.0.0.0/16 -j ACCEPT # PostgreSQL

I’m not a big fan of iptables-persistent (and friends) and configure my whole firewall setup in /etc/rc.local. Obviously you can use your existing iptables tooling to accomplish this task.

Adapt Xen default configuration

To shorten xen-create-image calls I’ve added all my default settings to /etc/xen-tools/xen-tools.conf:

lvm = vg0
Use volume group vg0 for logical volumes (guest image and swap). If you’ve no free space in your volume group just use loopback images and remove this option.
size = 4G
Small default setting. Often increased on command line
memory = 128M
Small default setting. Often increased on command line
fs = ext4
ext4 instead of default ext3
genpass_len = 20
Password length of random root password
boot = 1
Boot guests after creation
output = /etc/xen/guests You need to create /etc/xen/guests
nohosts = 1
Don’t add new guests to dom0’s /etc/hosts file. I use DNS for name resolution.

Add custom roles

xen-create-image has the option --roles which is a great opportunity to apply final customations after creating the guest. Just put a shell script to /etc/xen-tools/role.d/ (or use the existing ones if they fit your needs) and add the name to the --role parameter when creating a new image.
I use a custom role called codingminds to configure my basic environment settings (ZSH, options, prompt, etc.), add some default programs and adjust the OS for my setup:

#!/bin/sh

prefix=$1

#
#  Source our common functions
#
if [ -e /usr/share/xen-tools/common.sh ]; then
    . /usr/share/xen-tools/common.sh
else
    . ./hooks/common.sh
fi

#
# Log our start
#
logMessage Script $0 starting

#
# SSH
#
if mkdir -p ${prefix:?}/root/.ssh && chmod 0700 ${prefix:?}/root/.ssh; then
    logMessage "successfully generated /root/.ssh directory"
else
    logMessage "failed to generate /root/.ssh directory"
fi
if cat /root/.ssh/*.pub >> ${prefix:?}/root/.ssh/authorized_keys && chmod ; then
    logMessage "successfully added public keys"
else
    logMessage "failed to add public keys"
fi

#
# Basic packages
#
installDebianPackage ${prefix} nano htop lsof strace tcpdump

#
# Unattended upgrades
#
cat <<EOF > ${prefix:?}/etc/apt/apt.conf.d/20auto-upgrades
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
EOF
if [ $? -eq 0 ]; then
    logMessage "successfully set /etc/apt/apt.conf.d/20auto-upgrades"
else
    logMessage "failed to set /etc/apt/apt.conf.d/20auto-upgrades"
fi

cat <<"EOF" > ${prefix:?}/etc/apt/apt.conf.d/50unattended-upgrades
Unattended-Upgrade::Allowed-Origins {
  "${distro_id}:${distro_codename}";
  "${distro_id}:${distro_codename}-security";
  "${distro_id}ESM:${distro_codename}";
  "${distro_id}:${distro_codename}-updates";
  "${distro_id}:${distro_codename}-proposed";
  "${distro_id}:${distro_codename}-backports";
  "TorProject:${distro_codename}";
};
Unattended-Upgrade::Package-Blacklist {
//  "libc6-i686";
};
Unattended-Upgrade::AutoFixInterruptedDpkg "true";
Unattended-Upgrade::MinimalSteps "true";
Unattended-Upgrade::Mail "technik@coding-minds.com";
Unattended-Upgrade::MailOnlyOnError "true";
Unattended-Upgrade::Remove-Unused-Dependencies "true";
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-Time "05:00";
EOF
if [ $? -eq 0 ]; then
    logMessage "successfully set /etc/apt/apt.conf.d/50unattended-upgrades"
else
    logMessage "failed to set /etc/apt/apt.conf.d/50unattended-upgrades"
fi

#
# Fix resolve.conf
#
if cp /etc/resolv.conf ${prefix:?}/etc/resolvconf/resolv.conf.d/base; then
    logMessage "successfully set resolvconf base"
else
    logMessage "failed to set resolvconf base"
fi

#
# TZ
#

cat <<EOF >> ${prefix:?}/etc/environment
TZ=:/etc/localtime
EOF
if [ $? -eq 0 ]; then
    logMessage "successfully set TZ environment"
else
    logMessage "failed to set TZ environment"
fi

#
# ZSH
#

installDebianPackage ${prefix} zsh

cat <<"EOF" > ${prefix:?}/root/.zshrc
[ -f ~/.profile ] && source ~/.profile
# Prompt
export PS1='%F{cyan}%m:%~ $ %f'

# Behaviour
zstyle ':completion:*' rehash true
set -o emacs

# Exports
export PATH=${PATH}:${HOME}/.bin

# Default editor
export EDITOR=nano

# Command completion
autoload -Uz compinit
compinit
setopt COMPLETE_ALIASES
fpath=(~/.zsh $fpath)

# History
setopt hist_expire_dups_first
setopt hist_ignore_dups
setopt extended_history
setopt share_history
SAVEHIST=1000000000
HISTSIZE=2147483647
HISTFILE=~/.zsh_history

autoload -U +X bashcompinit && bashcompinit
EOF
if [ $? -eq 0 ]; then
    logMessage "successfully set .zshrc"
else
    logMessage "failed to set .zshrc"
fi

if chroot ${prefix:?} /usr/bin/chsh -s /usr/bin/zsh root; then
    logMessage "successfully set ZSH for root"
else
    logMessage "failed to set ZSH for root"
fi

## Postfix

installDebianPackage ${prefix} postfix

sed -i -e '/relayhost/d' -e '/inet_interfaces/d' ${prefix:?}/etc/postfix/main.cf && \
cat <<EOF >> ${prefix:?}/etc/postfix/main.cf
relayhost = mail.coding-minds.com
inet_interfaces = loopback-only
EOF
if [ $? -eq 0 ]; then
    logMessage "successfully set /etc/postfix/main.cf"
else
    logMessage "failed to set /etc/postfix/main.cf"
fi

cat <<EOF > ${prefix:?}/etc/aliases
root: technik@coding-minds.com
EOF
if [ $? -eq 0 ]; then
    logMessage "successfully set /etc/aliases"
else
    logMessage "failed to set /etc/aliases"
fi

if chroot ${prefix:?} /usr/sbin/postalias /etc/aliases; then
    logMessage "successfully updated aliases"
else
    logMessage "failed to update aliases"
fi

#
# Disable IPv6 :(
#

cat <<EOF >>${prefix:?}/etc/sysctl.conf
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
EOF
if [ $? -eq 0 ]; then
    logMessage "successfully disabled IPv6"
else
    logMessage "failed to disable IPv6"
fi

#
# Log our finish
#
logMessage Script $0 finished

Adapt BIOS settings

There is one big issue with the current Xen version on Ubuntu Xenial and AMD Ryzen 7 mainboards used by Hetzner (B350M PRO-VDH). As a default setting the BIOS of those mainboards has Core Performance Boost and AMD Cool'n'Quiet activated.
At least one of those two settings result in kernel freezes if the host is not utilized much. That’s funny to debug because if you are on the system and monitor dmesg, journalctl and other stuff there’s enough workload to not trigger the effect of those settings. It just freezes short after you stopped looking at it.
You’ll find similar bug reports on Xen mailing lists if you search for something like

[101279.189190] INFO: rcu_sched detected stalls on CPUs/tasks:
[101279.189295]         0-...: (3 GPs behind) idle=1be/140000000000000/0 softirq=787339/787339 fqs=2285 
[101279.189406]         (detected by 14, t=15002 jiffies, g=815643, c=815642, q=47)
[101279.189500] Sending NMI from CPU 14 to CPUs 0:

Since disabling this BIOS settings the freeze didn’t happen again. Save BIOS settings

Create/manage/destroy a guest

That’s pretty basic and well documented in man pages. Therefore this is more a short list of well known commands than a real explanation.

Create

All default settings in /etc/xen-tools/xen-tools.conf can be overruled by commandline arguments:
xen-create-image --role=codingminds --dist=xenial --gateway=10.0.0.1 --netmask=255.255.0.0 --memory=4gb --vcpus=4 --hostname=your.vm.fqdn.tld --ip=10.0.0.8 --size=10G

The command above will create a new guest image with Ubuntu Xenial, 4 assigned vCPUs, 4GB RAM, 10GB virtual disk and 256MB swap (defined in /etc/xen-tools/xen-tools.conf). The VM has the hostname your.vm.fqdn.tld and listens on IP 10.0.0.8/16.

Stop and start

xl create /etc/xen/guests/your.vm.fqdn.tld.cfg
Start an existing guest image
xl shutdown your.vm.fqdn.tld
Send a shutdown signal to the guest OS
xl reboot your.vm.fqdn.tld
Send a reboot signal to the guest OS
xl destroy your.vm.fqdn.tld
Destroy the guest immediately (e.g. if guest OS is stalled)

Delete

To remove a guest (and free the used disk space) delete the guest image with xen-delete-image your.vm.fqdn.tld --lvm=vg0. After that the configuration file in /etc/xen/guests/ is still there, you have to delete it manually.

Autostart

Xen has also some sort of autostart for guests. Just link the configuration files from /etc/xen/guests/ to /etc/xen/auto/. I’m not really happy with this auto start thing because I’ve noticed Xen often failes to start those hosts if there are more then just a handful of guests. As a consequence I’ll build a simple systemd unit to handle this requirement.

About

Blog

Privacy

Ubuntu Xenial Xen on Hetzner