Ceph recover a RBD image from a dead cluster

Many years ago I came across a script made by Shawn Moore and Rodney Rymer from Catawba university. The purpose of this tool is to reconstruct a RBD image. Imagine your cluster dead, all the monitors got wiped off and you don’t have backup (I know what can possibly happen?). However all your objects remain intact.

I’ve always wanted to blog about this tool, simply to advocate it and make sure that people can use it. Hopefully it will be a good publicity for this tool :-).

Backuping RBD images

Before we dive into the recovery process. I’d like to take a few lines to describe what is important to backup and how to backup it.

Keep track of all the images across all the pools
Store their properties (shown by rbd info <pool>/<image>)
Store the RBD headers

Recover

In the context of this exercise I will simply:

Create a RBD image
Map it on a machine
Put a XFS filesystem on top of it
Touch a simple file

$ rbd create -s 10240 leseb
$ rbd info leseb
rbd image 'leseb':
  size 10240 MB in 2560 objects
  order 22 (4096 kB objects)
  block_name_prefix: rb.0.1066.74b0dc51
  format: 1

$ sudo rbd -p rbd map leseb
/dev/rbd0

$ sudo rbd showmapped
id pool image snap device
0  rbd  leseb -    /dev/rbd0

$ sudo mkfs.xfs /dev/rbd0
log stripe unit (4194304 bytes) is too large (maximum is 256KiB)
log stripe unit adjusted to 32KiB
meta-data=/dev/rbd0              isize=256    agcount=17, agsize=162816 blks
         =                       sectsz=512   attr=2, projid32bit=0
data     =                       bsize=4096   blocks=2621440, imaxpct=25
         =                       sunit=1024   swidth=1024 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=8 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

$ sudo mount /dev/rbd0 /mnt
$ echo "foo" > /mnt/bar
$ sudo umount /mnt
$ sudo rbd unmap /dev/rbd0

Prepare a directory on a server to restore your image:

$ mkdir recover_leseb
$ wget -O rbd_restore https://raw.githubusercontent.com/smmoore/ceph/master/rbd_restore.sh
$ chmox +x rbd_restore

Then I need to collect all the RBD object files, on my setup I only had one OSD server, which made the gathering operation easier:

$ cd recover_leseb
~/recover_leseb$ for block in $(find /var/lib/ceph/osd/ -type f -name  rb.0.1066.74b0dc51.*); do cp $block . ; done
~/recover_leseb$ bash recover.sh leseb rb.0.1066.74b0dc51 10737418240
~/recover_leseb$ file leseb
leseb: SGI XFS filesystem data (blksz 4096, inosz 256, v2 dirs)
~/recover_leseb$ du -h leseb
11M leseb

Hum looks like we have something interesting here :) Let’s see if it really worked:

~/recover_leseb$ losetup -f
/dev/loop0

~/recover_leseb$ losetup /dev/loop0 leseb
~/recover_leseb$ mount /dev/loop0 /mnt/
~/recover_leseb$ df -h /mnt
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0       10G   33M   10G   1% /mnt

~/recover_leseb$ ls /mnt/
bar
~/recover_leseb$ cat /mnt/bar
foo

HELL YEAH!