Ceph

Tomorrow, the first conference fully dedicated to Ceph will start in Beijing, China. I’m attending and super excited. I will see you there!

Learning Ceph - Second Edition was published in October 2017. This is special post to highlight a new book I’ve been helping with. Good colleagues of mine wrote that book and I encourage anyone …

A couple of releases ago, in order to minimize changes within the ceph.conf.j2 Jinja template, we introduced a new module that we took from the OpenStack Ansible guy. This module is called …

Thanks to this recent pull request, you can now bootstrap the Ceph Manager daemon. This new daemon was added during the Kraken development cycle, its main goal is to act as a hook for existing system …

During the last CDM (Ceph Developer Monthly), I presented a blueprint that will help Ceph playing nicely when it’s being containerized.

I’ve been getting a lot of questions about the RBD mirroring daemon so I thought I will do a blog post similar to a FAQ. Most of the features described in this article will likely be released …

Just like promised last Monday, this article is the first of a series of informative blog posts about incoming Ceph features. Today, I’m cheating a little bit because I will decrypt one …

Happy New year! Bonne Année ! Best wishes to my readers :). C’est le turfu and Ceph is moving fast, really fast and you won’t believe how many awesome features are currently in the pipe. …

I guess you got lucky, or maybe I felt so bad not posting anything for more than a month but here it is the last blog post of the year :). With the latest release of Ceph, Jewel, a new Rados Gateway …

Long time no see! I know, though times at the office, this won’t probably stop until Christmas break, so I’ll do my best to keep up the pace. In this article, I will explain how to take …

This article is co-authored with Gregory Charot (author of the tool). Have you ever found yourself doing long series of pipes to get a particular value that is not directly provided by a Ceph CLI …

For those of you who are not familiar with the concept yet, I’m going to give a quick introduction on Hyperconverged infrastructure.

Quick tip on how to configure SSL with Civetweb on Rados Gateway.

The new Ubuntu LTS 16.04, code name: Xenial Xerus was just released a couple of weeks ago. Interestingly the new Ceph LTS, code name Jewel also just got released! Being a really lover of Ansible, I …

As presented in my preview of BlueStore, this new store has the ability tp be configured with multiple devices. Since the ceph-disk utility does not support configuring multiple devices, OSD must be …

Another feature preview for Jewel. NBD driver for RBD that allows librbd to present a kernel-level block device

The RBD mirroring feature will be available for the next stable Ceph release: Jewel.

A new way to efficiently store objects using BlueStore.

Get the modification time of a RBD image.

Following last week article, here is another CRUSH example. This time we want to store our first replica on SSD drives and the second copy on SATA drives.

Quick CRUSH example on how to store 3 replicas, two in rack number 1 and the third one in rack number 2.

Sometimes removing OSD, if not done properly can result in double rebalancing. The best practice to remove an OSD involves changing the crush weight to 0.0 as first step.

Ceph just moved outside of DevStack in order to comply with the new DevStack’s plugin policy. The code can be found on github. We now have the chance to be on OpenStack Gerrit as well and thus …

When you manage a large cluster, you do not always know where your OSD are located. Sometimes you have issues with PG such as unclean or with OSDs such as slow requests. While looking at your ceph …

Infernalis has just been released a couple of weeks ago and I have to admit that I am really impressed of the work that has been done. So I am going to present you 5 really handy things that came out …

Quick tip to release the memory that tcmalloc has allocated but which is not being used by the Ceph daemon itself.

This article simply relays some recent discovery made around Ceph performance. The finding behind this story is one of the biggest improvement in Ceph performance that has been seen in years. So I …

Quick and simple test to validate if the RBD cache is enabled on your client.

Quick tip to determine the location of a file stored on CephFS.

Using SSD drives in some part of your cluster might useful. Specially under read oriented workloads. Ceph has a mechanism called primary affinity, which allows you to put a higher affinity on your …

The title is probably weird and misleading but I could not find better than this :). The idea here is to dive a little bit into what the kernel client sees for each client that has a RBD device …

The Hammer release brought the support of a new feature for RBD images called object map. The object map tracks which blocks of the image are actually allocated and where. This is especially useful …

Quick tip.

RBD readahead was introduced with Giant.

Quick tip. Simply check the diff between the applied configuration in your ceph.conf and the default values on an OSD.

Ceph monitors make use of leveldb to store cluster maps, users and keys. Since the store is present, Ceph developers thought about exposing this through the monitors interface. So monitors have a …

Debugging scrubbing errors can be tricky and you don’t necessary know how to proceed.

This is a quick note about Ceph networks, so do not expect anything lengthy here :). Usually Ceph networks are presented as cluster public and cluster private. However it is never mentioned that you …

Simple trick to analyse the write patterns applied to your Ceph journal.

Recently I improved a playbook that I wrote a couple of months ago regarding Ceph rolling upgrades. This playbook is part of the Ceph Ansible repository and available as rolling_update.yml …

Analyse IO pattern of all your guest machines.

Recently I built a little repository on github/ceph where I put two files to help you building your DevStack Ceph. $ git clone https://git.openstack.org/openstack-dev/devstack $ git clone …

Many years ago I came across a script made by Shawn Moore and Rodney Rymer from Catawba university. The purpose of this tool is to reconstruct a RBD image. Imagine your cluster dead, all the monitors …

Space reclamation mechanism for the Kernel RBD module. Having this kind of support is really crucial for operators and ease your capacity planing. RBD images are sparse, thus size after creation is …

OSD performance counters tend to stack up and sometimes the value shown is not really representative of the current environment. Thus it is quite useful to reset the counters to get the last values. …

While playing with Ceph on DevStack I noticed that after several rebuild I ended up with the following error from nova-scheduler: Secret not found: rbd no secret matches uuid …

Quick tip to collect Kernel RBD logs.

Quick tip on how to retrieve cache statistics from the a cache pool.

Introducing the ability to connect DevStack to a remote Ceph cluster. So DevStack won’t bootstrap any Ceph cluster, it will simply connect to a remote one.

A common recommendation is to store OSD journal on a SSD drive which implies loosing your OSD if this journal fails. This article assumes that your OSDs have been originally deployed with ceph-disk. …

During some strange circonstances, the levelDB monitor store can start taking up a substancial amount of space. Let quickly see how we can workaround that.

This second edition of the CephDay London looks really promising. You should definitely look at the agenda! Talks go from OpenStack to deep performance studies and crossing CephFS news! Check this …

A simple benchmark job to determine if your SSD is suitable to act as a journal device for your OSDs.

Almost two years ago, I was writing the results of the experiments with Flashcache. Today this blog post is a featured post, this howto was written by Andrei Mikhailovsky. Thanks for his contribution …

Features for the seventh Ceph release (Giant) have been frozen 3 weeks ago. Thus Giant is just around the corners and bugs are currently being fixed. This article is a quick preview on a new feature.

Useful to understand benchmark result and Ceph’s second write penalty (this phenomena is explained here in the section I.1).

The use case is simple, I want to use both SSD disks and SATA disks within the same machine and ultimately create pools pointing to SSD or SATA disks. In order to achieve our goal, we need to modify …

A couple a months ago, Dan Mick posted a nice article that introduced the RBD support for iSCSI / TGT. In this article, I will have a look at it.

Quick tip to enable the dynamic subtree tree partitionning with multiple Ceph MDS servers.

Moving further on the Software Defined Storage principles, Ceph, with its latest stable version introduced a new mechanism called cache pool tiering. It brings a really interesting concept that will …

A simple script to bootstrap a Ceph cluster and start playing with it. The script heavily relies on: Vagrant Ceph Ansible The final machine will contain: 1 Monitor 3 OSDs 1 MDS 1 RADOS Gateway

For quite some time, Ceph has an admin API. This article demonstrates and gives some hints to monitor Ceph.

Following up this article. This playbook was made to automate Ceph servers maintenance. The typical use case is an hardware change. By running this playbook you will set the noout flag on your …

A simple in-memory backend that stores objects in RAM is available since version 0.76 and will land with Firefly. Quick look at this new feature.

It has been two weeks now since we released the Ansible playbook for Ceph. This article is a little update concerning new features and roadmap.

Ceph: cow based deployment with Ansible :-).

A quick post to tell everybody that I’ll be giving a presentation at the Ceph day in Frankfurt. Obviously, I’ll be representing the company I work for, eNovance. The event is next …

Ceph IO patterns analysis final part: The Ugly.

Ceph IO patterns analysis part 2: The Bad.

Ceph IO patterns analysis part 1: The Good.

Quick tip to perform a full snapshot of an RBD image. (only MtG players can recognize this picture!)

As you may know Ceph is not part of Debian Wheezy, thus QEMU-KVM was not compiled with the Ceph support (--enable-rbd with both librbd and librados). This article is just a quick tip to detect if …

Getting more familiar with the Ceph CLI with CRUSH.

Ceph admin API init script.

RBD images are thin-provisionned thus you don’t always know the real size of the image. Moreover, Ceph doesn’t provide any simple facility to check the real size of an image. This blog …

Some experiment with gigantic overprovisioned RBD images.

Some figures from a RADOS bench.

The Ceph developer summit is already behind us and wow! so many good things are around the corner! During this online event, we discussed the future of the Firefly release (planned for February …

Memory leaks disappeared and CPU load dramatically reduced. Yay!

Curious? Wanna know who has a RBD device mapped?

Quick how-to on mapping/unmapping a RBD device during startup and shutdown.

Quick script to evaluate the placement of the objects contained in a RBD image. #!/bin/bash # USAGE # ./rbd-loc <pool> <image> if [ -z ${1} ] || [ -z ${2} ]; then echo "USAGE: …

The goal of this little analysis was to determine the overhead generated by Ceph. One important point was also to estimate the deviance brought by Ceph between RAW IOs from disk and Ceph IOs.

A quick post to tell everybody that I’ll be giving a presentation at the Ceph day in London. Obviously, I’ll be representing the company I work for, eNovance. The event is next Wednesday, …

Today the CoreOS team released its first OpenStack image. Let’s quickly see how we can take advantage of it.

By default, OpenStack doesn’t use any caching. However, you might want to enable the RBD caching.

I simply collected all my localrc files and combined them into a single one. Feel free to grab the parts that interest you.

DevStack in 1 minute! Ready, steady, GO!

It’s not really clear from the command line

The road to Havana is long, no milestone in the corner yet, but already some enhancements have been brought to the Horizon interface. Let’s take a quick look at the new fancy stuff!

What a perfect picture, a Cephalopod smoking a cigar! Updates! The OpenStack developer summit was great and obviously one of the most exciting session was the one about the Ceph integration with …

Quick Nova API CLI updates.

Archiving deleted rows made easy by Grizzly.

How-to quickly deploy a MDS server.

The title of the article is not that explicit, actually I had trouble to find a proper one. Thus let me clarify a bit. Here is the context I was wondering if Glance was capable of converting images …

Grizzly brought the multi-backend functionality to cinder and tons of new drivers. The main purpose of this article is to demonstrate how we can take advantage of the tiering capability of Ceph.

Materials to start playing with Ceph. This Vagrant box contains a all-in-one Ceph installation.

Sometimes it’s just funny to experiment the theory, just to notice “oh well it works as expected”. This is why today I’d like to share some experiments with 2 really specific …

Quite recently François Charlier and I worked together on the Puppet modules for Ceph on behalf of our employer eNovance. In fact, François started to work on them last summer, back then he achieved …

Short short update.

A Placement Group (PG) aggregates a series of objects into a group, and maps the group to a series of OSDs. A common mistake while creating a pool is to use the rados command which by default creates …

The title of the article is a bit wrong, but it’s certainly the easiest to understand :-).

It’s fair to say that the geo-replication is one of the most requested feature by the community. This article is draft, a PoC about Ceph geo-replication. Disclaimer: yes this setup is tricky …

How to use a memory profiler to track memory usage of Ceph daemons!

A lot of new features came with the version 0.55 of Ceph, one of them is that CephX authentication is enable by default. If you run v0.48 Argonaut without CephX and want to update to the latest …

Configure logging in Ceph.

Several way to know where your instances run.

Every once in a while you really want to cleanup the token table of the Keystone database. A couple of weeks ago while backuping my cloud controller I noticed that the backup of the Keystone database …

This article introduces a simple use case for storage providers. For some reasons some customers would like to pay more for a fast storage solution and other would prefer to pay less for a …

I’ve been often asked a couple of time, why CephFS is not ready? Does this statment is still true?

In this article, I will introduce the Block device cacher concept and the different solutions available. I will also (because it’s the title of the article) explain how to increase the …

Couple of days ago I was looking for a way to lock an instance but something more powerful than a simple lock via the nova API.

Folsom brought a lot of new features, however while playing with nova-manage, I came across a new option, option that I found very useful and that I’m about to share with you.

I would like to share with you readers some of the optimizations I made on the SSDs storing my Ceph Journals. This article has been placed into the ceph categorie, but it’s more general best …

Usually, you don’t always want to restart your daemon everytime you change your configuration. Fortunatly, Ceph supports parameter injection!

Quick overview about placement groups within Ceph.

Little update of my previous script for Folsom.

It’s been a while that I heard that mkcephfs was deprecated, but for the benefit of which tool? I have named ceph-deploy. The project is young and it’s work in progress but at a …

Folsom has been released, it’s probably time for some of you to deploy OpenStack. This is a follow up to the article titled From nova-network to Quantum. One of the main question with Folsom …

Before starting anything I’d like highlight the fact that this article was co-written with Emilien Macchi, who has a better expertise on Quantum that I do. In this article, we focused on both …

Inktank’s guys are awesome!

Quick and short introduction to a well known bug in nova-volume. I simply want to point it out because I think it could be useful.

Brief analysis of the ceph -s command :)

The time has come to perform some benchmark with Ceph. You expect (or not), they are there, my Ceph’s benchmarks!

Quick tips about how to manage a production environement. A simple use case here put your ceph journal on a SSD on a production cluster while clients are writting.

The release of the first stable version of Ceph brought a lot of improvment at admin socket level. This is a really useful command that you should use.

If you want to change or rebuild a new set of monitor servers, you will need to manipulate the monmap. This monmap is exchange between every monitors.

While strolling within the ceph mailing list, I came accross something really useful, something that I was looking for :)

I never thought about blogging this, but I think it can be useful for those of you who don’t know this little tool called: Multitail.

Bring gigabit to your VM’s NIC!

A little bit more about RBD objects…

Quick script to delete all the intances in error state.

Since CephFS is not most mature component in Ceph, you won’t consider to use it on a production platform. In this article, I offer a possible solution to expose RBD to a shared filesystem.

Quick tip. If you don’t use CephFS, you don’t need a MDS server. However if you installed it simply to try CephFS and if you’r done playing with it here is how to delete the MDS …

Quick tip. If you want to download a glance image stored in a RBD backend.

Quick tip on Ceph. The linux kernel RBD (rados block device) driver allows striping a linux block device over multiple distributed object store data objects. The libceph module takes care of that.

Little study about the recent release of CloudStack as Apache Licence. Little battle between OpenStack and CloudStack, FIGHT!