Fencing LXC containers with STONITH
Introducing fencing on LXC container.
Introduction
I.1. About LXC
LXC or Linux Container is an amazing way to run multiple environments on a single machine. No virtualisation here, only several environment sharing the same Kernel. LXC is similar to OpenVZ. LXC is a young project but it grows really quickly and it’s well maintained by Canonical. LXC is also part of the Ubuntu Cloud Stack. LXC is not as mature as his elder but it’s coming. This formadible technology has a lot of use case, often close to the virtualisation ones but the hudge benefit of container based virtualisation is the performance. You can barely notice the performance difference between the host and the container. Thus yes, LXC is a real use case for production and is not stuck to testing environment. A simple use case could be the following: you want or have plained to build an HA stack, a classical one: LAMP but you only have 2 servers… Basically in this context 2 servers mean one dedicated for the frontend Apache and one for the database MySQL backend. Thanks to LXC you could start with 4 containers and start to build an HA stack like so:
Host 1:
- Apache 1
- MySQL Master 1
Host 2:
- Apache 2
- MySQL Master 2
When you will have more budget, you will invest in 2 new servers, copy your existing containers into your new servers, re-start your container et voilà! you have a real HA stack in a minute thanks to LXC. LXC is really handy for managing and backuping containers. It’s also really flexible in term of CPU/RAM allocation and disk quota.
This was only one use case, but you could also want to run multiple version of a given software. Let’s say you want to run MySQL 5.3 and 5.5 but you don’t really want to buy 4 servers for that (HA is the watchword) maybe for several reasons:
- Your application doesn’t require so much power
- You don’t have the budget for 4 servers
In this case, LXC becomes tremendously useful :).
I.2. About the STONITH death match
One thing that I hate to do is paraphrasing existing content. The Web is full of excellent articles about STONITH, so I will simply give you a little reminder. Shoot The Other Node In The Head is the mechanism which fences node (s). It’s essentially use to tell the cluster about the state of given resource. Several scenario may end up to a split brain situation, this will require the usage of STONITH. For instance a network communication issues, the node can communicate with the rest of the cluster.
At the end, keep in mind that STONITH is equivalent to this situation:
Some good reading:
II. Setup libvirt with LXC
First manage your LXC with libvirt, install it:
$ sudo apt-get install libvirt-bin -y |
According to your current configuration create xml for your container.
Here is my current container, the one used from my previous article:
$ ls /var/lib/lxc/ |
The most important directory is rootfs. Now define define your libvirt domain like so:
<domain type='lxc'> |
You can now start up your container like so:
$ sudo virsh -c lxc:/// define mon-conteneur.xml |
Finally run it!
$ sudo virsh -c lxc:/// start mon-conteneur |
Perform the same operation for your second container. At the end you should have something like this:
$ sudo virsh list |
Laziness! By default the virsh uri is qemu:///system, virsh looks at an environment variable to set it:
export VIRSH_DEFAULT_CONNECT_URI=lxc:///
You can verify with:
$ virsh uri |
To get inside your container you have 2 options:
- tty console
- SSH
Try with the console:
$ sudo virsh -c lxc:/// console mon-conteneur |
By SSH, but first find the IP address given by dnsmasq:
$ arp -an | grep virbr0 |
III. Pacemaker fencing
III.1. Quick pacemaker installation
Install pacemaker within each container:
$ sudo apt-get install pacemaker -y |
Quickly edit your corosync.conf and modify the bindnetadd with the subnet of your container. Since your containers are manage by libvirt (and if you didn’t specify anything else), they will get an IP address from the dnsmasq instance affiliate to libvirt-bin. So basically you should use the configuration below:
interface {
# The following values need to be set based on your environment
ringnumber: 0
bindnetaddr: 192.168.122.0
mcastaddr: 226.94.1.1
mcastport: 5405
}
Finally start corosync on both containers:
$ sudo service corosync start |
III.2. Fencing prerequisite
Configure every machine (host and containers) and modify the libvirt behavior by editing your /etc/libvirt/libvirtd.conf. For the containers you will need to install the package libvirt-bin too and perform the same operation as above:
listen_tls = 0
listen_tcp = 1
tcp_port = "16509"
auth_tcp = "none"
And your /etc/default/libvirtd-bin with:
libvirtd_opts="-d -l"
Finally restart libvirt-bin and verify that libvirt is listenning :). Also don’t forget to allow the containers to connect to the host:
$ sudo iptables -I INPUT -m tcp -p tcp --dport 16509 -j ACCEPT |
You should be able to run this test from one of your container:
$ sudo virsh --connect=lxc+tcp://192.168.0.1 list --all |
Here the IP 192.168.0.1 is the address of the host machine.
III.3. STONITH resources
Configure your stonith resources:
$ sudo crm configure primitive p_fence_mon_conteneur stonith:external/libvirt \ |
$ sudo crm configure primitive p_fence_mon_conteneur-2 stonith:external/libvirt \ |
Please notice that
hypervisor_uriis the IP address of the host running my containers. Driver (lxc) + Transport (tcp) + IPhostlistthe name of the container to fence (use the name shown byvirsh list)
Modify these values according to your setup.
We keep away each resource from their own server. Location rules are set like this because by convention STONITH doesn’t fence his own node.
$ sudo crm configure location loc_fench_node1 fence_mon_conteneur p_fence_mon_conteneur -inf: mon-conteneur |
You should end up we something like this:
============
Last updated: Sat Sep 1 05:48:37 2012
Last change: Sat Sep 1 05:48:34 2012 via cibadmin on container
Stack: openais
Current DC: mon-conteneur-2 - partition with quorum
Version: 1.1.6-9971ebba4494012a93c03b40a2c58ec0eb60f50c
2 Nodes configured, 2 expected votes
2 Resources configured.
============
Online: [ mon-conteneur-2 mon-conteneur ]
p_fence_mon_conteneur (stonith:fence_virsh): Started mon-conteneur-2
p_fence_mon_conteneur-2 (stonith:fence_virsh): Started mon-conteneur
IV. Shoot!
From one of your container you can easily simulate a fencing operation:
$ sudo stonith -t external/libvirt hostlist=mon-conteneur hypervisor_uri="lxc+tcp://192.168.146.195" -T off mon-conteneur |
Perform a reboot:
$ sudo stonith -t external/libvirt hostlist=mon-conteneur hypervisor_uri="lxc+tcp://192.168.146.195" -T reset mon-conteneur |
If you want to get more information about this STONITH plugin:
$ sudo stonith -m -t external/libvirt |
Et voilà! This article didn’t aim to do an LXC how-to but if you want to go further you can visit the Ubuntu documentation. LXC is a growing technology which involves more and more production use case. Hope it helps!
Comments