Quick tip on Ceph. The linux kernel RBD (rados block device) driver allows striping a linux block device over multiple distributed object store data objects. The libceph module takes care of that.
Warning: Don’t use rbd kernel driver on the osd server. Perhaps it will freeze the rbd client and your osd server.
$ sudo apt-get install ceph-common
$ sudo modprobe rbd
Starting with rbd:
$ lsmod rbd
rbd 28158 1
libceph 129934 2 rbd,ceph
$ rbd create share --size 4096
$ rbd ls
share
With Cephx:
$ ceph-authtool --print-key /etc/ceph/keyring.admin
AQDVGc5P0LXzJhAA5C019tbdrgypFNXUpG2cqQ==
$ sudo echo "172.17.1.4:6789,172.17.1.5:6789,172.17.1.7:6789 name=admin,secret=AQDVGc5P0LXzJhAA5C019tbdrgypFNXUpG2cqQ== rbd share" | sudo tee /sys/bus/rbd/add
Here rbd is the default pool and share is the name of my device. You can easily get more information about your device, it should identified with the name 0:
$ ls /sys/bus/rbd/devices/0/
client_id create_snap current_snap major name pool power refresh size subsystem uevent
$ grep "" /sys/bus/rbd/devices/0/*
/sys/bus/rbd/devices/0/client_id:client5279
grep: /sys/bus/rbd/devices/0/create_snap: Permission denied
/sys/bus/rbd/devices/0/current_snap:-
/sys/bus/rbd/devices/0/major:251
/sys/bus/rbd/devices/0/name:share
/sys/bus/rbd/devices/0/pool:rbd
grep: /sys/bus/rbd/devices/0/refresh: Permission denied
/sys/bus/rbd/devices/0/size:4294967296
/sys/bus/rbd/devices/0/uevent:DEVTYPE=rbd
A /dev/ device should have been generated, now format the devive and put a filesystem on it:
$ sudo mkfs.xfs /dev/rbd0
$ sudo mount /dev/rbd0 /mnt/
$ df -h | grep rbd
/dev/rbd0 4.0G 33M 4.0G 1% /mnt
Now you can use it as a simple device or put NFS on it.
/!\ Warning: make sure that this bug does not affect your Ceph version before trying to remove a RBD device.
This article was a simple introduction to the RBD device. Since I don’t want to paraphrase the Ceph wiki, if you need more information consult it the official wiki.