As part of my work on ceph-docker, I have been playing a bit with Kubernetes. As a first exercise, I thought it would be interesting to run my Ceph monitors with Kubernetes. Let’s see how we can achieve that.
The following assumes that you have a Kubernetes cluster up and running.
$ wget https://raw.githubusercontent.com/ceph/ceph-docker/master/examples/kubernetes/ceph-mon.json
$ kubectl create -f mon.json
replicationControllers/ceph-mon
$ kubectl get pod
POD IP CONTAINER(S) IMAGE(S) HOST LABELS STATUS CREATED MESSAGE
ceph-mon-qo4yc 127.0.0.1/ name=frontend Pending 3 seconds
ceph-mon ceph/daemon
ceph-mon-sv56v 127.0.0.1/ name=frontend Pending 3 seconds
ceph-mon ceph/daemon
ceph-mon-zc0lb 127.0.0.1/ name=frontend Pending 3 seconds
ceph-mon ceph/daemon
$ kubectl log ceph-mon-qo4yc
creating /etc/ceph/ceph.client.admin.keyring
creating /etc/ceph/ceph.mon.keyring
creating /var/lib/ceph/bootstrap-osd/ceph.keyring
creating /var/lib/ceph/bootstrap-mds/ceph.keyring
creating /var/lib/ceph/bootstrap-rgw/ceph.keyring
monmaptool: monmap file /etc/ceph/monmap
monmaptool: set fsid to ef67b6f1-8d50-49fe-a8dd-c81bd7dd09c2
monmaptool: writing epoch 0 to /etc/ceph/monmap (1 monitors)
creating /tmp/ceph.mon.keyring
importing contents of /etc/ceph/ceph.client.admin.keyring into /tmp/ceph.mon.keyring
importing contents of /var/lib/ceph/bootstrap-osd/ceph.keyring into /tmp/ceph.mon.keyring
importing contents of /var/lib/ceph/bootstrap-mds/ceph.keyring into /tmp/ceph.mon.keyring
importing contents of /var/lib/ceph/bootstrap-rgw/ceph.keyring into /tmp/ceph.mon.keyring
importing contents of /etc/ceph/ceph.mon.keyring into /tmp/ceph.mon.keyring
ceph-mon: set fsid to b2beb954-6791-408c-8212-9559fd5daf36
ceph-mon: created monfs at /var/lib/ceph/mon/ceph-ceph-mon-qo4yc for mon.ceph-mon-qo4yc
2015-06-18 09:29:11.347368 7f4924a168c0 0 ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3), process ceph-mon, pid 1
2015-06-18 09:29:11.717765 7f4924a168c0 0 mon.ceph-mon-qo4yc does not exist in monmap, will attempt to join an existing cluster
2015-06-18 09:29:11.718113 7f4924a168c0 0 using public_addr 172.17.0.11:6789/0 -> 172.17.0.11:6789/0
2015-06-18 09:29:11.718225 7f4924a168c0 0 starting mon.ceph-mon-qo4yc rank -1 at 172.17.0.11:6789/0 mon_data /var/lib/ceph/mon/ceph-ceph-mon-qo4yc fsid b2beb954-6791-408c-8212-9
559fd5daf36
...
...
...
...
It seems to run fine, now let’s have a look at the Ceph status:
$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1f86ffbd3897 ceph/daemon:latest "/entrypoint.sh" 18 seconds ago Up 17 seconds k8s_ceph-mon.b9e89a17_ceph-mon-zc0lb_de
fault_76b6b559-159c-11e5-9d52-fa163ebdff5b_895951f7
a0f966c5cd54 ceph/daemon:latest "/entrypoint.sh" 26 seconds ago Up 23 seconds k8s_ceph-mon.b9e89a17_ceph-mon-qo4yc_de
fault_76b6eefc-159c-11e5-9d52-fa163ebdff5b_cea0ff5e
a9fa1663d712 ceph/daemon:latest "/entrypoint.sh" 26 seconds ago Up 24 seconds k8s_ceph-mon.b9e89a17_ceph-mon-sv56v_de
fault_76b71555-159c-11e5-9d52-fa163ebdff5b_8d5b1f56
fceabf9c1b77 gcr.io/google_containers/pause:0.8.0 "/pause" 28 seconds ago Up 26 seconds k8s_POD.e4cc795_ceph-mon-sv56v_default_
76b71555-159c-11e5-9d52-fa163ebdff5b_998441ff
5f9eecd36fb5 gcr.io/google_containers/pause:0.8.0 "/pause" 28 seconds ago Up 26 seconds k8s_POD.e4cc795_ceph-mon-qo4yc_default_
76b6eefc-159c-11e5-9d52-fa163ebdff5b_3af31e82
c99221438e93 gcr.io/google_containers/pause:0.8.0 "/pause" 28 seconds ago Up 26 seconds k8s_POD.e4cc795_ceph-mon-zc0lb_default_
76b6b559-159c-11e5-9d52-fa163ebdff5b_dfc6b697
$ sudo docker exec 1f86ffbd3897 ceph -s
cluster b2beb954-6791-408c-8212-9559fd5daf36
health HEALTH_ERR
64 pgs stuck inactive
64 pgs stuck unclean
no osds
monmap e3: 3 mons at {ceph-mon-qo4yc=172.17.0.11:6789/0,ceph-mon-sv56v=172.17.0.12:6789/0,ceph-mon-zc0lb=172.17.0.10:6789/0}
election epoch 8, quorum 0,1,2 ceph-mon-zc0lb,ceph-mon-qo4yc,ceph-mon-sv56v
osdmap e1: 0 osds: 0 up, 0 in
pgmap v2: 64 pgs, 1 pools, 0 bytes data, 0 objects
0 kB used, 0 kB / 0 kB avail
64 creating
As we can see, all the three monitors successfully formed a quorum.