Ceph on Sébastien Han

Rook just landed on operatorhub.io

Fri, 17 May 2019 22:59:04 +0000

I’m excited to announce that just right for Cephalocon and KubeCon Europe, Rook has landed on operatorhub.io. It has been quite a challenge to have it merged, but in the end my pull request got merged :). If you want to know what this means for upstream you should look at this article.

OpenStack and Ceph for Distributed Hyperconverged Edge Deployments

Tue, 12 Mar 2019 16:42:48 +0000

I’m simply relaying an article I reviewed and helped writting. It is reflecting my talk from the last OpenStack Summit in Berlin.

You can read it here Thanks to the author for capturing the essence of the talk.

Ceph nano is getting better and better

Sun, 24 Feb 2019 14:42:07 +0000

Long time no blog, I know, I know… Soon, I will do another blog entry to “explain” a little why I am not blogging as much I used too but if you’re still around and reading this then thank you! For the past few months, cn has grown in functionality so let’s explore what’s new and what’s coming.

Ceph Nano big updates

Mon, 30 Apr 2018 12:36:11 +0000

With its two latest versions (v1.3.0 and v1.4.0) Ceph Nano brought some nifty new functionalities that I’d like to highlight in the article.

Ansible module to manage CephX Keys

Mon, 02 Apr 2018 00:31:25 +0000

Following our recent initiative on writing more Ceph modules for Ceph Ansible, I’d like to introduce one that I recently wrote: ceph_key.

See you at the first Cephalocon

Tue, 20 Mar 2018 18:52:12 +0000

Tomorrow, the first conference fully dedicated to Ceph will start in Beijing, China. I’m attending and super excited. I will see you there!

Huge changes in ceph-container

Mon, 19 Mar 2018 00:19:07 +0000

A massive refactor done a week ago on ceph-container. And yes, I’m saying ceph-container, not ceph-docker anymore. We don’t have anything against Docker, we believe it’s excellent and we use it extensively. However, having the ceph-docker name does not reflect the content of the repository. Docker is only the Dockerfile, the rest is either entrypoints or examples. In the end, we believe ceph-container is a better match for the repository name.

Ansible module to create CRUSH hierarchy

Thu, 15 Mar 2018 22:32:25 +0000

First post of the year after a long time with no article, three months… I know it has been a while, I wish I had more time to do more blogging. I have tons of draft articles that never made it through, I need to make up for lost time.

So for this first post, let me introduce an Ansible I wrote for ceph-ansible: ceph_crush.

Learning Ceph - Second Edition

Fri, 15 Dec 2017 10:35:02 +0000

Learning Ceph - Second Edition was published in October 2017.

This is special post to highlight a new book I’ve been helping with. Good colleagues of mine wrote that book and I encourage anyone willing to learn Ceph to get a copy of it. The book is available on Amazon.

Introducing ceph-nano

Tue, 28 Nov 2017 19:23:14 +0000

I’ve recently started a small project that aims to help developers working with the S3 API. The program is called cn for Ceph Nano, is available on github let me give you a tour of what it does. I initially presented the program during my talk at the last OpenStack summit in Sydney. Originally, I wrote a quick prototype in Bash, then moved to Go.

Introducing Ceph Ansible profile library

Mon, 27 Nov 2017 17:14:57 +0000

A couple of releases ago, in order to minimize changes within the ceph.conf.j2 Jinja template, we introduced a new module that we took from the OpenStack Ansible guy. This module is called config_template and allows us to declare Ceph configuration options as variables in your group_vars files. This is extremely useful for us

Based on that work and as part of the big ceph-ansible 3.0 release we added a profile directory that guides users on how to properly inject new configuration options. All of that is based on use cases. For instance, we currently have profile examples for configuring Ceph Rados Gateway with OpenStack Keystone.

Here is the current list of profiles:

rgw-keystone-v2
rgw-keystone-v3
rgw-radosgw-static-website
rgw-usage-log

More are coming and we expect to get more during the lifetime of the project. One particular profile that we might create is a performance oriented one when running Bluestore on NVMe drives.

The new Ceph container demo is super dope!

Tue, 27 Jun 2017 16:41:53 +0000

I have been recently working on refactoring our Ceph container images. We used to have two separate images for daemon and demo. Recently, for Luminous, I decided to merge the demo container into daemon. It makes everything easier, code is in a single place, we only have a single image to test with the CI and users have a single image to play with.

OpenStack Cinder configure replication API with Ceph

Mon, 19 Jun 2017 10:36:06 +0000

I just figured out that there hasn’t been much coverage on that functionality, even though we presented it last year at the OpenStack Summit.

Ceph Docker better support for Bluestore

Wed, 14 Jun 2017 11:16:04 +0000

I have been extensively working on ceph-docker for the last few months and it’s getting better. With the upcoming arrival of Ceph Luminous (next LTS), Bluestore will be the default backend to store objects. Thus, I had to spend some time working on improving the support for Bluestore.

Disabling scenarios in ceph-docker

Mon, 17 Apr 2017 10:35:34 +0000

I recently completed a full resync from Kraken to Jewel in ceph-docker in which I introduced a new feature to disable scenarios. Running an application on bleeding edge technology can be tough and challenging for individuals and also for companies. Even me, as a developer and for bleeding edge testers I’m tempted to release unstable features (understand not recommended for production). So sometimes it’s handy to have the ability to restrict the use of a software by disabling some of its functionality. This is exactly what I did for ceph-docker, in this article I’ll explain how that works.

Test Ceph Luminous pre-release with ceph-docker

Thu, 13 Apr 2017 11:55:17 +0000

*/!\ DISCLAIMER /!*

*/!\ DO NOT GET TOO EXCITED, AT THE TIME OF THE WRITTING LUMINOUS IS NOT OFFICIALLY RELEASE IN STABLE YET /!*

*/!\ USE AT YOUR OWN RISK, DO NOT PUT PRODUCTION DATA ON THIS /!*

Luminous is just around the corner but we have been having packages available for a couple of weeks already. That’s why I recently thought: “how come don’t we have any Ceph container image for Luminous then?”. And I know a lot of you are eager to test the latest developments of Bluestore (the new method to store objects, directly on a raw device).

Now it’s done, you can fetch the ceph/daemon image using one of these two tags:

tag-build-master-luminous-centos-7
tag-build-master-luminous-ubuntu-16.04

And you will get a running Ceph cluster on Luminous.

Ceph manager support in ceph-ansible and ceph-docker

Mon, 10 Apr 2017 11:21:22 +0000

Thanks to this recent pull request, you can now bootstrap the Ceph Manager daemon. This new daemon was added during the Kraken development cycle, its main goal is to act as a hook for existing system to get monitoring information from the Ceph cluster. It normally runs alongside monitor daemons but can be deployed to any particular node. Using your preferred method, you can deploy it in a non-containerized or containerized fashion with ceph-ansible.

Also, we just released a new tag for ceph-ansible, 2.2.0, we will go through heavy testing during the next couple of weeks. This will result in a new stable version branched on stable-2.2. I will a new blog post for stable-2.2 once it’s out to highlight some of the best features and functionality we added.

Debug Ceph Docker containers

Mon, 27 Mar 2017 17:00:49 +0000

For the last couple of weeks, Erwan Velu and I have been busy refactoring the entire ceph-docker code base. Through these cosmetic changes, we implemented new mechanisms to finely grained debug containers at run time. This article was co-written with Erwan Velu and will explain what we worked on.

Containerize Ceph: store config options in monitor KV store

Mon, 27 Feb 2017 14:26:36 +0000

During the last CDM (Ceph Developer Monthly), I presented a blueprint that will help Ceph playing nicely when it’s being containerized.

No more privileged containers for Ceph OSDs

Mon, 20 Feb 2017 15:45:50 +0000

I’m really sorry for being so quiet lately, I know I promised to release articles more regularly and I clearly failed… Many things are going on and where motivation is key to write articles, I’ve been having a hard time to find the right motivation to write :/

However, I am not giving up and I finally found the time to write a little bit on the things we improved in ceph-docker, our Ceph in container project.

Ceph and RBD mirroring, upcoming enhancements

Thu, 16 Feb 2017 17:50:01 +0000

I’ve been getting a lot of questions about the RBD mirroring daemon so I thought I will do a blog post similar to a FAQ. Most of the features described in this article will likely be released for Ceph Luminous. Luminous should land this spring, so be patient :).

Ceph RBD and iSCSI

Thu, 05 Jan 2017 18:12:10 +0000

Just like promised last Monday, this article is the first of a series of informative blog posts about incoming Ceph features. Today, I’m cheating a little bit because I will decrypt one particular feature that went a bit unnoticed with Jewel. So we are discussing something that is already available but will have follow-ups with new Ceph releases. The feature doesn’t really have a name but it’s along the line of having an iSCSI support with the RBD protocol. With that, we can connect Ceph storage to hypervisors and/or operating systems that don’t have a native Ceph support but understand iSCSI. Technically speaking this targets non-Linux users who can not use librbd with QEMU or krbd directly.

Ceph, the future of storage, incoming features blog series

Mon, 02 Jan 2017 11:52:50 +0000

Happy New year! Bonne Année ! Best wishes to my readers :).

C’est le turfu and Ceph is moving fast, really fast and you won’t believe how many awesome features are currently in the pipe. So to start the year off the wheels, I’m planning on publishing a set of articles to tease you a little bit. But don’t get to excited, this is on-going work that will mature next year only. Starting this week on Friday and all the following ones and for an undetermined period of time. See you this Friday for the first blog post!

Ceph Rados Gateway and NFS

Fri, 23 Dec 2016 10:20:05 +0000

I guess you got lucky, or maybe I felt so bad not posting anything for more than a month but here it is the last blog post of the year :).

With the latest release of Ceph, Jewel, a new Rados Gateway feature came out. This feature hasn’t really been advertised yet so I thought I will do a blog post. This is an initial implementation that will be improved in the first releases of Ceph obviously.

As this requires a couple of components, it is quite difficult at the moment to get it easily working. So even if we support it in ceph-ansible, this is not that stable yet. For example on Ubuntu, we need it to ship Ceph v10.2.5 on Xenial so that nfs-ganesha 2.4 can build a working Rados Gateway FSAL.

Ceph ansible is building its community

Mon, 05 Dec 2016 15:58:34 +0000

This blog just relays what the initial announcement of the ceph-ansible mailing list.

Run multiple Rados Gateways on the same host with Ceph Ansible

Mon, 07 Nov 2016 18:29:47 +0000

Quick how-to with Ceph Ansible to run multiple Ceph Rados Gateways on the same machine.

Configure OpenStack Glance for RBD mirroring

Wed, 02 Nov 2016 15:42:17 +0000

Since Ceph Jewel, we have the RBD mirroring functionality and people have been starting using it for multi-site and disaster recovery use cases. The tool is not perfect but is rock solid, expect many enhancements in the future release such as support for multiple peer and daemons. From a pure OpenStack perspective, to enable this feature we don’t really want to add any code into Glance Store. The reason is simple, glance’s store code looks up for specific Ceph features into the Ceph configuration file itself. So there is no point of adding a new configuration flag into Glance that says something like enable_image_journaling. The operator will only have to configure Ceph, that’s it.

Devstack Ceph supports containerized Ceph

Fri, 21 Oct 2016 15:18:20 +0000

Yes people, I’m still alive :). As you might noticed, I’ve been having a hard time to keep up the pace with blogging. It’s mainly due to me traveling a lot these days and preparing conferences. It’s a really busy end of the year for me :).

Fortunately, I’m still finding the time to work on some new features to projects I like. As you might know, I’ve been busy working on ceph-ansible and ceph-docker, trying conciliate both and making sure they work well together. In ceph-docker, we have an interesting container image, that I already presented here. I was recently thinking we could use it to simplify the Ceph bootstrapping process in DevStack. The patch I recently merge doesn’t get ride of the “old” way to bootstrap, the path is just a new addition, a new deployment method.

In practice, this doesn’t change anything for me, but at some point it allows us to validate that a containerized Ceph doesn’t have any problem and bring the same functionality as a non-containerized Ceph. Without further ado, let’s jump into this!

Ceph migrate from non-containerized to containerized daemons

Mon, 26 Sep 2016 11:34:57 +0000

Long time no see! I know, though times at the office, this won’t probably stop until Christmas break, so I’ll do my best to keep up the pace. In this article, I will explain how to take over an existing Ceph bare metal deployment and use containers. Spoiler: Ansible baby, Ansible!

Ceph ansible can now shrink your cluster

Tue, 16 Aug 2016 14:23:20 +0000

Ceph ansible is quickly catching up with ceph-deploy in terms of features. Last week, I was discussing the dm-crypt support. The ability to shrink a Ceph cluster, removing one or N monitors/OSDs wasn’t possible until very recently. Let’s have a look at this new feature.

Introducing ceph-lazy

Thu, 28 Jul 2016 16:58:29 +0000

This article is co-authored with Gregory Charot (author of the tool). Have you ever found yourself doing long series of pipes to get a particular value that is not directly provided by a Ceph CLI command or just trying to remove surrounding text to get a particular value? This situation often results in quick & dirty sed/awk pipelines ending (best case scenario) as alias or forgotten in your shell history until next time you need it. Here comes ceph-lazy, a shell toolkit that combines some of these queries that require multiple processing or text manipulation.

Ceph zap device container

Mon, 25 Jul 2016 11:24:43 +0000

Some use cases might require to zap a device (destroy partition tables) prior to run your Ceph OSD container with a dedicated disk. While running development environment this is particularly interesting as this allows us to quickly bootstrap and tear down sandboxes.

Ceph RBD mirror daemon available in containers

Thu, 21 Jul 2016 10:39:15 +0000

I recently pushed into ceph-docker the support for thr RBD mirror. This daemon is responsable for asynchronously replicating RBD images from one cluster to another. The main purpose of the daemon is to address disaster recovery use cases.

Quick dive into hyperconverged architecture with OpenStack and Ceph

Mon, 11 Jul 2016 14:16:40 +0000

For those of you who are not familiar with the concept yet, I’m going to give a quick introduction on Hyperconverged infrastructure.

Busy working on ceph-docker :)

Mon, 06 Jun 2016 20:10:38 +0000

I have been having a hard time keeping up with all things happening at the office and the blog. One of the main reason is that I actively resumed my work on ceph-docker, in this article I will explain some of the things I have been working on.

Configure Rados Gateway Civetweb with SSL

Wed, 25 May 2016 14:26:51 +0000

Quick tip on how to configure SSL with Civetweb on Rados Gateway.

Using Ubuntu? Planning on using Ceph Jewel? Here what you should consider

Wed, 18 May 2016 23:49:14 +0000

The new Ubuntu LTS 16.04, code name: Xenial Xerus was just released a couple of weeks ago. Interestingly the new Ceph LTS, code name Jewel also just got released! Being a really lover of Ansible, I just got what I would call an interesting idea.

The OpenStack Ceph Galaxy

Mon, 16 May 2016 14:15:02 +0000

Picture of our galaxy :). This picture describes the state of the integration of Ceph into OpenStack.

OpenStack Cinder: discard support for Ceph in Mitaka

Wed, 11 May 2016 14:49:20 +0000

OpenStack Mitaka brought the support of a new feature. This feature is a follow-up of the Nova discard implementation. Now it is possible to configure Cinder per backend.

Bootstrap two Ceph and configure RBD mirror using Ceph Ansible

Mon, 09 May 2016 17:04:34 +0000

Since Jewel is out everyone wants to try the new RBD-mirroring feature.

Ceph Jewel: configure BlueStore with multiple devices

Wed, 04 May 2016 15:25:35 +0000

As presented in my preview of BlueStore, this new store has the ability tp be configured with multiple devices. Since the ceph-disk utility does not support configuring multiple devices, OSD must be configured manually. Let’s see how we can configure this.

Ceph Jewel Preview: map RBD devices on NBD

Mon, 04 Apr 2016 14:31:00 +0000

Another feature preview for Jewel. NBD driver for RBD that allows librbd to present a kernel-level block device

Ceph Jewel Preview: Ceph RBD mirroring

Mon, 28 Mar 2016 10:43:00 +0000

The RBD mirroring feature will be available for the next stable Ceph release: Jewel.

Ceph Jewel Preview: a new store is coming, BlueStore

Mon, 21 Mar 2016 10:19:00 +0000

A new way to efficiently store objects using BlueStore.

Ceph: Modification Time of RBD Images

Mon, 28 Dec 2015 15:27:00 +0000

Get the modification time of a RBD image.

Ceph CRUSH rule: 1 copy SSD and 1 copy SATA

Mon, 21 Dec 2015 13:38:00 +0000

Following last week article, here is another CRUSH example. This time we want to store our first replica on SSD drives and the second copy on SATA drives.

Ceph CRUSH two copies in one rack

Mon, 14 Dec 2015 15:35:00 +0000

Quick CRUSH example on how to store 3 replicas, two in rack number 1 and the third one in rack number 2.

Ceph: properly remove an OSD

Fri, 11 Dec 2015 15:21:00 +0000

Sometimes removing OSD, if not done properly can result in double rebalancing. The best practice to remove an OSD involves changing the crush weight to 0.0 as first step.

Ceph is moving outside DevStack core to a plugin

Mon, 30 Nov 2015 14:55:00 +0000

Ceph just moved outside of DevStack in order to comply with the new DevStack’s plugin policy. The code can be found on github. We now have the chance to be on OpenStack Gerrit as well and thus brings all the good things from the OpenStack infra (a CI).

To use it simply create a localrc file with the following:

enable_plugin ceph https://github.com/openstack/devstack-plugin-ceph

A more complete localrc file can be found on Github.

Ceph: find an OSD location and restart it

Fri, 27 Nov 2015 14:54:00 +0000

When you manage a large cluster, you do not always know where your OSD are located. Sometimes you have issues with PG such as unclean or with OSDs such as slow requests. While looking at your ceph health detail you only see where the PGs are acting or on which OSD you have slow requests. Given that you might have tons of OSDs located on a lot of node, it is not straightforward to find and restart them.

Five useful new features from Ceph Infernalis

Wed, 18 Nov 2015 16:38:00 +0000

Infernalis has just been released a couple of weeks ago and I have to admit that I am really impressed of the work that has been done. So I am going to present you 5 really handy things that came out with this new release.

Ceph: release RAM used by TCMalloc

Wed, 09 Sep 2015 15:06:00 +0000

Quick tip to release the memory that tcmalloc has allocated but which is not being used by the Ceph daemon itself.

The Ceph and TCMalloc performance story

Mon, 07 Sep 2015 15:08:00 +0000

This article simply relays some recent discovery made around Ceph performance. The finding behind this story is one of the biggest improvement in Ceph performance that has been seen in years. So I will just highlight and summarize the study in case you do not want to read it entirely.

Ceph: validate that the RBD cache is active

Wed, 02 Sep 2015 15:16:00 +0000

Quick and simple test to validate if the RBD cache is enabled on your client.

CephFS: determine a file location

Mon, 31 Aug 2015 13:43:00 +0000

Quick tip to determine the location of a file stored on CephFS.

Ceph: get the best of your SSD with primary affinity

Thu, 06 Aug 2015 18:14:00 +0000

Using SSD drives in some part of your cluster might useful. Specially under read oriented workloads. Ceph has a mechanism called primary affinity, which allows you to put a higher affinity on your OSDs so they will likely be primary on some PGs. The idea is to have reads served by the SSDs so clients can get faster reads.

Deploy Ceph on Docker with Ansible

Fri, 31 Jul 2015 10:33:00 +0000

Deploy Ceph on bare metal with Ansible

Mon, 20 Jul 2015 11:20:00 +0000

See what the Ceph client sees

Wed, 08 Jul 2015 10:32:00 +0000

The title is probably weird and misleading but I could not find better than this :). The idea here is to dive a little bit into what the kernel client sees for each client that has a RBD device mapped. In this article, we are focusing on the Kernel RBD feature.

Ceph enable the object map feature

Mon, 06 Jul 2015 14:11:00 +0000

The Hammer release brought the support of a new feature for RBD images called object map. The object map tracks which blocks of the image are actually allocated and where. This is especially useful for operations on clones like resize, import, export, flattening and size calculation since the client does not need to calculate where each object is located. The client will just look up on that table.

Ceph is complaining: too many PGs

Tue, 16 Jun 2015 11:29:00 +0000

Quick tip.

Ceph: activate RBD readahead

Wed, 03 Jun 2015 10:40:00 +0000

RBD readahead was introduced with Giant.

Ceph OSD daemon config diff

Mon, 01 Jun 2015 14:38:00 +0000

Quick tip. Simply check the diff between the applied configuration in your ceph.conf and the default values on an OSD.

Ceph using Monitor key/value store

Mon, 04 May 2015 00:03:00 +0000

Ceph monitors make use of leveldb to store cluster maps, users and keys. Since the store is present, Ceph developers thought about exposing this through the monitors interface. So monitors have a built-in capability that allows you to store blobs of data in a key/value fashion. This feature has been around for quite some time now (something like 2 years), but haven’t got any particular attention since then. I even noticed that I never blogged about it :).

Ceph: manually repair object

Mon, 27 Apr 2015 16:09:00 +0000

Debugging scrubbing errors can be tricky and you don’t necessary know how to proceed.

Stretching Ceph networks

Thu, 16 Apr 2015 23:12:00 +0000

This is a quick note about Ceph networks, so do not expect anything lengthy here :).

Usually Ceph networks are presented as cluster public and cluster private. However it is never mentioned that you can use a separate network for the monitors. This might sound obvious for some people but it is completely possible. The only requirement of course is to have this monitor network accessible from all the Ceph nodes.

Ceph: analyse journal write pattern

Mon, 13 Apr 2015 00:31:00 +0000

Simple trick to analyse the write patterns applied to your Ceph journal.

Ceph rolling upgrades with Ansible

Mon, 30 Mar 2015 15:47:00 +0000

Recently I improved a playbook that I wrote a couple of months ago regarding Ceph rolling upgrades. This playbook is part of the Ceph Ansible repository and available as rolling_update.yml Let’s have a look at it.

Analyse OpenStack guest writes and reads running on Ceph

Fri, 27 Feb 2015 15:24:00 +0000

Analyse IO pattern of all your guest machines.

Quick and efficient Ceph DevStacking

Tue, 24 Feb 2015 11:27:00 +0000

Recently I built a little repository on github/ceph where I put two files to help you building your DevStack Ceph.

$ git clone https://git.openstack.org/openstack-dev/devstack
$ git clone https://github.com/ceph/ceph-devstack.git
$ cp ceph-devstack/local* devstack
$ cd devstack
$ ./stack.sh

Happy DevStacking!

Ceph recover a RBD image from a dead cluster

Thu, 29 Jan 2015 14:35:00 +0000

Many years ago I came across a script made by Shawn Moore and Rodney Rymer from Catawba university. The purpose of this tool is to reconstruct a RBD image. Imagine your cluster dead, all the monitors got wiped off and you don’t have backup (I know what can possibly happen?). However all your objects remain intact.

I’ve always wanted to blog about this tool, simply to advocate it and make sure that people can use it. Hopefully it will be a good publicity for this tool :-).

Ceph and KRBD discard

Mon, 26 Jan 2015 00:01:00 +0000

Space reclamation mechanism for the Kernel RBD module. Having this kind of support is really crucial for operators and ease your capacity planing. RBD images are sparse, thus size after creation is equal to 0 MB. The main issue with sparse images is that images grow to eventually reach their entire size. The thing is Ceph doesn’t know anything that this happening on top of that block especially if you have a filesystem. You can easily write the entire filesystem and then delete everything, Ceph will still believe that the block is fully used and will keep that metric. However thanks to the discard support on the block device, the filesystem can send discard flush commands to the block. In the end, the storage will free up blocks.

Ceph reset perf counter

Fri, 16 Jan 2015 11:18:00 +0000

OSD performance counters tend to stack up and sometimes the value shown is not really representative of the current environment. Thus it is quite useful to reset the counters to get the last values. This feature was added in the Ceph 0.90, so you must wait for the Hammer release.

This action can be triggered via the admin socket:

$ sudo ceph daemon osd.0 perf reset

Fix nova-scheduler issue with RBD and UUID not found

Wed, 14 Jan 2015 17:43:00 +0000

While playing with Ceph on DevStack I noticed that after several rebuild I ended up with the following error from nova-scheduler:

Secret not found: rbd no secret matches uuid '3092b632-4e9f-40ca-9430-bbf60cefae36'

Actually this error is reported by libvirt itself which somehow keeps the secret in-memory (I believe) even when a new virsh secret is applied. The only solution I have found so far to this issue is to restart libvirt:

$ sudo service libvirt-bin restart

Ceph: collect Kernel RBD logs

Wed, 17 Dec 2014 20:29:00 +0000

Quick tip to collect Kernel RBD logs.

Ceph: Cache Tier Statistics

Mon, 15 Dec 2014 12:20:00 +0000

Quick tip on how to retrieve cache statistics from the a cache pool.

DevStack and remote Ceph cluster

Thu, 11 Dec 2014 23:05:00 +0000

Introducing the ability to connect DevStack to a remote Ceph cluster. So DevStack won’t bootstrap any Ceph cluster, it will simply connect to a remote one.

Ceph: recover OSDs after SSD journal failure

Thu, 27 Nov 2014 11:05:00 +0000

A common recommendation is to store OSD journal on a SSD drive which implies loosing your OSD if this journal fails. This article assumes that your OSDs have been originally deployed with ceph-disk. You will also realise that it’s really simple to bring your OSDs back to life after replacing your faulty SSD with a new one.

Ceph: monitor store taking up a lot of space

Mon, 27 Oct 2014 15:31:00 +0000

During some strange circonstances, the levelDB monitor store can start taking up a substancial amount of space. Let quickly see how we can workaround that.

See you at the CephDay London 2!

Tue, 14 Oct 2014 17:33:00 +0000

This second edition of the CephDay London looks really promising. You should definitely look at the agenda! Talks go from OpenStack to deep performance studies and crossing CephFS news!

Check this out on the Eventbrite page.

I hope to see you there! I don’t have any talks, at least for once I’ll be watching :-).

Ceph: how to test if your SSD is suitable as a journal device?

Fri, 10 Oct 2014 12:12:00 +0000

A simple benchmark job to determine if your SSD is suitable to act as a journal device for your OSDs.

Ceph and Enhanceio

Mon, 06 Oct 2014 00:01:00 +0000

Almost two years ago, I was writing the results of the experiments with Flashcache. Today this blog post is a featured post, this howto was written by Andrei Mikhailovsky. Thanks for his contribution to this blog :-).

Ceph: RBD import and export get parallelized in Giant

Sun, 28 Sep 2014 17:01:00 +0000

Features for the seventh Ceph release (Giant) have been frozen 3 weeks ago. Thus Giant is just around the corners and bugs are currently being fixed. This article is a quick preview on a new feature.

Analyse Ceph object directory mapping on disk

Mon, 08 Sep 2014 16:52:00 +0000

Useful to understand benchmark result and Ceph’s second write penalty (this phenomena is explained here in the section I.1).

Ceph: mix SATA and SSD within the same box

Mon, 25 Aug 2014 16:52:00 +0000

The use case is simple, I want to use both SSD disks and SATA disks within the same machine and ultimately create pools pointing to SSD or SATA disks. In order to achieve our goal, we need to modify the CRUSH map. My example has 2 SATA disks and 2 SSD disks on each host and I have 3 hosts in total.

Start with the RBD support for TGT

Mon, 07 Jul 2014 15:59:00 +0000

A couple a months ago, Dan Mick posted a nice article that introduced the RBD support for iSCSI / TGT. In this article, I will have a look at it.

UNSTABLE: test the dynamic tree partitionning with multiple Ceph MDS

Wed, 02 Jul 2014 17:00:00 +0000

Quick tip to enable the dynamic subtree tree partitionning with multiple Ceph MDS servers.

Ceph cache pool tiering: a scalable and distributed cache

Tue, 10 Jun 2014 09:18:00 +0000

Moving further on the Software Defined Storage principles, Ceph, with its latest stable version introduced a new mechanism called cache pool tiering. It brings a really interesting concept that will help us to provide scalable distributed caching.

Vagrant up: install Ceph in one command

Thu, 01 May 2014 01:52:00 +0000

A simple script to bootstrap a Ceph cluster and start playing with it. The script heavily relies on:

The final machine will contain:

1 Monitor
3 OSDs
1 MDS
1 RADOS Gateway

Ceph: monitoring with the Ceph Admin API

Mon, 14 Apr 2014 15:32:00 +0000

For quite some time, Ceph has an admin API. This article demonstrates and gives some hints to monitor Ceph.

Ceph maintenance with Ansible

Wed, 09 Apr 2014 16:46:00 +0000

Following up this article.

This playbook was made to automate Ceph servers maintenance. The typical use case is an hardware change. By running this playbook you will set the noout flag on your cluster, which means that OSD can’t be marked as out of the CRUSH map, but they will be marked as down. Thus the OSD will not receive any data. Basically we tell the cluster to do not move any data since the operation will not last for too long.

Ceph: real men use the memstore backend

Thu, 03 Apr 2014 23:48:00 +0000

A simple in-memory backend that stores objects in RAM is available since version 0.76 and will land with Firefly. Quick look at this new feature.

Ansible Ceph playbook updates

Mon, 17 Mar 2014 14:38:00 +0000

It has been two weeks now since we released the Ansible playbook for Ceph. This article is a little update concerning new features and roadmap.

Announcing Ansible playbooks for Ceph!

Mon, 03 Mar 2014 16:06:00 +0000

Ceph: cow based deployment with Ansible :-).

See you at the Cephdays in Frankfurt

Mon, 24 Feb 2014 21:34:00 +0000

A quick post to tell everybody that I’ll be giving a presentation at the Ceph day in Frankfurt. Obviously, I’ll be representing the company I work for, eNovance. The event is next Thursday, February 27, 2014. I am going to talk about Ceph, performance and benchmarking. You can check the details and schedule of the event here. See you there!

Ceph IO patterns: the ugly

Mon, 24 Feb 2014 16:21:00 +0000

Ceph IO patterns analysis final part: The Ugly.

Ceph IO patterns: the bad

Mon, 17 Feb 2014 16:21:00 +0000

Ceph IO patterns analysis part 2: The Bad.

Ceph IO patterns: the good

Mon, 10 Feb 2014 11:36:00 +0000

Ceph IO patterns analysis part 1: The Good.

Ceph: snapshot a RBD instance with QEMU-UTILS

Mon, 20 Jan 2014 11:06:00 +0000

Quick tip to perform a full snapshot of an RBD image. (only MtG players can recognize this picture!)

Check if KVM has the support of Ceph

Fri, 17 Jan 2014 10:26:00 +0000

As you may know Ceph is not part of Debian Wheezy, thus QEMU-KVM was not compiled with the Ceph support (--enable-rbd with both librbd and librados). This article is just a quick tip to detect if your QEMU-KVM has the support of Ceph.

Ceph: managing CRUSH with the CLI

Mon, 13 Jan 2014 10:49:00 +0000

Getting more familiar with the Ceph CLI with CRUSH.

Ceph admin API init script

Wed, 08 Jan 2014 09:50:00 +0000

Ceph admin API init script.

Real size of a Ceph RBD image

Thu, 19 Dec 2013 10:49:00 +0000

RBD images are thin-provisionned thus you don’t always know the real size of the image. Moreover, Ceph doesn’t provide any simple facility to check the real size of an image. This blog post took his inspiration from the Ceph mailing list.

RBD image bigger than your Ceph cluster

Thu, 12 Dec 2013 11:35:00 +0000

Some experiment with gigantic overprovisioned RBD images.

Ceph RADOS benchmarks replica impacts

Mon, 09 Dec 2013 15:37:00 +0000

Some figures from a RADOS bench.

Ceph performance: interesting things going on

Mon, 02 Dec 2013 00:01:00 +0000

The Ceph developer summit is already behind us and wow! so many good things are around the corner! During this online event, we discussed the future of the Firefly release (planned for February 2014). During the last OpenStack summit in Hong Kong, I had the opportunity to discuss with Sage a new feature that might go into Firefly. This was obviously discussed during the CDS too. His plan is to add a multi-backend functionality for the filestore. And trust me this will definitely bring Ceph to another level.

Quick update Ceph: from Argonaut to Cuttlefish

Thu, 28 Nov 2013 00:01:00 +0000

Memory leaks disappeared and CPU load dramatically reduced. Yay!

Ceph: find who's mapping a RBD device

Mon, 25 Nov 2013 10:56:00 +0000

Curious? Wanna know who has a RBD device mapped?

Map/unmap RBD device on boot/shutdown

Fri, 22 Nov 2013 19:42:00 +0000

Quick how-to on mapping/unmapping a RBD device during startup and shutdown.

Ceph RBD objects placement

Tue, 19 Nov 2013 15:50:00 +0000

Quick script to evaluate the placement of the objects contained in a RBD image.

#!/bin/bash

# USAGE
# ./rbd-loc <pool> <image>

if [ -z ${1} ] || [ -z ${2} ];
then
 echo "USAGE: ./rbd-loc <pool> <image>"
 exit 1
fi

rbd_prefix=$(rbd -p ${1} info ${2} | grep block_name_prefix | awk '{print $2}')
for i in $(rados -p ${1} ls | grep ${rbd_prefix})
do
 ceph osd map ${1} ${i}
done

Quick analysis of the Ceph IO layer

Thu, 03 Oct 2013 00:36:00 +0000

The goal of this little analysis was to determine the overhead generated by Ceph. One important point was also to estimate the deviance brought by Ceph between RAW IOs from disk and Ceph IOs.

See you at the cephday in London

Tue, 01 Oct 2013 15:59:00 +0000

A quick post to tell everybody that I’ll be giving a presentation at the Ceph day in London. Obviously, I’ll be representing the company I work for, eNovance. The event is next Wednesday, October 9, 2013. I am going to talk about Ceph, performance and benchmarking. You can check the details and schedule of the event here. See you there!

Update: slides available through my employer’s blog (eNovance).

Want to save money in the Cloud? OpenStack CoreOS image is there

Fri, 30 Aug 2013 00:15:00 +0000

Today the CoreOS team released its first OpenStack image. Let’s quickly see how we can take advantage of it.

Configure Ceph RBD caching on OpenStack Nova

Thu, 22 Aug 2013 15:39:00 +0000

By default, OpenStack doesn’t use any caching. However, you might want to enable the RBD caching.

Best localrc for DevStack

Fri, 16 Aug 2013 21:35:00 +0000

I simply collected all my localrc files and combined them into a single one. Feel free to grab the parts that interest you.

DevStack in 1 minute

Thu, 08 Aug 2013 23:00:00 +0000

DevStack in 1 minute! Ready, steady, GO!

Ceph: update Cephx Keys

Fri, 26 Jul 2013 21:30:00 +0000

It’s not really clear from the command line

Some horizon enhancements

Mon, 17 Jun 2013 13:41:00 +0000

The road to Havana is long, no milestone in the corner yet, but already some enhancements have been brought to the Horizon interface. Let’s take a quick look at the new fancy stuff!

Ceph integration in OpenStack: Grizzly update and roadmap for Havana

Mon, 03 Jun 2013 15:11:00 +0000

What a perfect picture, a Cephalopod smoking a cigar! Updates! The OpenStack developer summit was great and obviously one of the most exciting session was the one about the Ceph integration with OpenStack. I had the great pleasure to attend this session (led by Josh Durgin from Inktank). The session was too short, but we had enough time to discuss upcoming features and establish a proper roadmap. With the help of this article, I’d like to share some of the discussions that we had and blueprints that we wrote.

Grizzly Nova: what's new in the API CLI?

Mon, 27 May 2013 21:58:00 +0000

Quick Nova API CLI updates.

Nova: archive deleted rows

Thu, 23 May 2013 22:38:00 +0000

Archiving deleted rows made easy by Grizzly.

Deploy a Ceph MDS server

Mon, 13 May 2013 16:15:00 +0000

How-to quickly deploy a MDS server.

Use existing RBD image and put it into Glance

Tue, 07 May 2013 17:15:00 +0000

The title of the article is not that explicit, actually I had trouble to find a proper one. Thus let me clarify a bit. Here is the context I was wondering if Glance was capable of converting images within its store. The quick answer is no, but I think such feature is worth to be implemented. Glance could be able to convert a QCOW2 image to a RAW format. Usually if you already have an image within let’s say a Ceph cluster (RBD), you have to download the image (since you probably don’t have the source image file anymore), then manually convert it with qemu-img (QCOW2 –> RAW) and eventually import it into Glance. Enough talk about this, I’ll address this in a future article. For now let’s stick to the first matter. Imagine that you have a KVM cluster backed by a Ceph Cluster and your CTO wants you to migrate the whole environment to OpenStack because it’s trendy (joking, OpenStack just rocks!). You’re not going to backup all your images and then build a new cluster or something like that, you might want OpenStack (Glance) to be aware of your Ceph cluster. Generally speaking you just have to connect Glance to one of your image pool. After this, the only thing to do is to create (it’s more registering the images ID and metadata than creating a new image) into Glance. No worries here’s the explanation. Longest introduction ever.

Ceph and Cinder multi-backend

Thu, 25 Apr 2013 12:03:00 +0000

Grizzly brought the multi-backend functionality to cinder and tons of new drivers. The main purpose of this article is to demonstrate how we can take advantage of the tiering capability of Ceph.

Play with Ceph - Vagrant Box

Mon, 22 Apr 2013 11:14:00 +0000

Materials to start playing with Ceph. This Vagrant box contains a all-in-one Ceph installation.

Some Ceph experiments

Wed, 17 Apr 2013 22:22:00 +0000

Sometimes it’s just funny to experiment the theory, just to notice “oh well it works as expected”. This is why today I’d like to share some experiments with 2 really specific flags: noout and nodown. Behaviors describe in the article are well known because of the design of Ceph, so don’t yell at me: ‘Tell us something we don’t know!’, simply see this article a set of exercises that demonstrate some Ceph internal functions :-).

Ceph Puppet Modules

Mon, 25 Mar 2013 15:58:00 +0000

Quite recently François Charlier and I worked together on the Puppet modules for Ceph on behalf of our employer eNovance. In fact, François started to work on them last summer, back then he achieved the Monitor manifests. So basically, we worked on the OSD manifest. Modules are in pretty good shape thus we thought it was important to communicate to the community. That’s enough talk, let’s dive into these modules and explain what do they do. See below what’s available:

Testing environment is Vagrant ready.
Bobtail Debian latest stable version will be installed
The module only supports CephX, at least for now
Generic deployment for 3 monitors based on a template file examples/common.sh which respectively includes mon.sh, osd.sh, mds.sh.
Generic deployment for N OSDs. OSD disks need to be set from the examples/site.pp file (line 71). Puppet will format specified disks in XFS (only filesystem implemented) using these options: -f -d agcount=<cpu-core-number> -l size=1024m -n size=64k and finally mounted with: rw,noatime,inode64. Then it will mount all of them and append the appropriate lines in the fstab file of each storage node. Finally the OSDs will be added into Ceph.

All the necessary materials (sources and how-to) are publicly available (and for free) under AGPL license on eNovance’s Github. Those manifests do the job quite nicely, although we still need to work on MDS (90% done, just need a validation), RGW (0% done) and a more flexible implementation (authentication and filesystem support). Obviously comments, constructive critics and feedback are more then welcome. Thus don’t hesitate to drop an email to either François (f.charlier@enovance.com) or I (sebastien@enovance.com) if you have further questions.

Grizzly availability zones

Mon, 18 Mar 2013 00:09:00 +0000

Short short update.

Ceph: change PG number on the fly

Tue, 12 Mar 2013 21:11:00 +0000

A Placement Group (PG) aggregates a series of objects into a group, and maps the group to a series of OSDs. A common mistake while creating a pool is to use the rados command which by default creates a pool of 8 PGs. Sometime you don’t properly know how to set this value thus you use the ceph command but put an extremely high value for it. Both case are bad and could lead to some unfortunate situations. In this article, I will explore some methods to work around this major problem.

Mount a specific pool with CephFS

Mon, 11 Feb 2013 12:20:00 +0000

The title of the article is a bit wrong, but it’s certainly the easiest to understand :-).

Ceph geo-replication (sort of)

Mon, 28 Jan 2013 20:09:00 +0000

It’s fair to say that the geo-replication is one of the most requested feature by the community. This article is draft, a PoC about Ceph geo-replication.

Disclaimer: yes this setup is tricky and I don’t guarantee that this will work for you.

Ceph and memory profiling

Thu, 17 Jan 2013 16:46:00 +0000

How to use a memory profiler to track memory usage of Ceph daemons!

Disable CephX for v0.55 and higher

Fri, 11 Jan 2013 01:34:00 +0000

A lot of new features came with the version 0.55 of Ceph, one of them is that CephX authentication is enable by default. If you run v0.48 Argonaut without CephX and want to update to the latest Bobtail, you might run through some problems if you don’t edit your configuration file.

Logging in Ceph

Mon, 07 Jan 2013 13:06:00 +0000

Configure logging in Ceph.

Where does my instance run?

Thu, 20 Dec 2012 13:17:00 +0000

Several way to know where your instances run.

Cleanup keystone tokens

Wed, 12 Dec 2012 19:56:00 +0000

Every once in a while you really want to cleanup the token table of the Keystone database. A couple of weeks ago while backuping my cloud controller I noticed that the backup of the Keystone database was longer than the other databases. After that, I checked the size of the dump (compressed) 60MB. Hummm but there is almost nothing in the Keystone database: users, tenants… wait.. could it be TOKENS?!

Ceph: manage storage zones with CRUSH

Fri, 07 Dec 2012 23:34:00 +0000

This article introduces a simple use case for storage providers. For some reasons some customers would like to pay more for a fast storage solution and other would prefer to pay less for a reasonnable storage solution.

Ceph and MDS

Mon, 03 Dec 2012 18:16:00 +0000

I’ve been often asked a couple of time, why CephFS is not ready? Does this statment is still true?

Make your RBD fly with flashcache

Thu, 15 Nov 2012 13:26:00 +0000

In this article, I will introduce the Block device cacher concept and the different solutions available. I will also (because it’s the title of the article) explain how to increase the performance of an RBD device with the help of flashcache in a Ceph cluster.

Tip: lock an instance against admin permissions

Wed, 07 Nov 2012 00:38:00 +0000

Couple of days ago I was looking for a way to lock an instance but something more powerful than a simple lock via the nova API.

Tip: Add a specific keypair to a flavor

Mon, 05 Nov 2012 22:12:00 +0000

Folsom brought a lot of new features, however while playing with nova-manage, I came across a new option, option that I found very useful and that I’m about to share with you.

Optimized your SSD

Mon, 29 Oct 2012 11:12:00 +0000

I would like to share with you readers some of the optimizations I made on the SSDs storing my Ceph Journals. This article has been placed into the ceph categorie, but it’s more general best practices SSDs.

Ceph: inject configuration without restart

Mon, 22 Oct 2012 19:15:00 +0000

Usually, you don’t always want to restart your daemon everytime you change your configuration. Fortunatly, Ceph supports parameter injection!

Ceph: placement groups

Mon, 15 Oct 2012 14:40:00 +0000

Quick overview about placement groups within Ceph.

Delete a VM in error state (Folsom)

Mon, 08 Oct 2012 17:26:00 +0000

Little update of my previous script for Folsom.

Introducing ceph-deploy

Tue, 02 Oct 2012 22:53:00 +0000

It’s been a while that I heard that mkcephfs was deprecated, but for the benefit of which tool? I have named ceph-deploy. The project is young and it’s work in progress but at a reasonnable advancement. Let’s get a first impression!

Quantum plugin comparison

Fri, 28 Sep 2012 11:23:00 +0000

Folsom has been released, it’s probably time for some of you to deploy OpenStack. This is a follow up to the article titled From nova-network to Quantum. One of the main question with Folsom is: which Quantum plugin should I use? The answer could be in this article! Another article co-written with Emilien Macchi. Deep dive into the available plugins in Quantum for OpenStack Folsom.

From nova-network to quantum

Tue, 25 Sep 2012 14:59:00 +0000

Before starting anything I’d like highlight the fact that this article was co-written with Emilien Macchi, who has a better expertise on Quantum that I do. In this article, we focused on both aspects of nova-network and quantum. We did not want to compare the two or similarly make them take each other head-on. This article is just a step back about the state of the networking in OpenStack. Basically what did we lose and what did we gain?

Inktank's guys are awesome

Thu, 13 Sep 2012 23:05:00 +0000

Inktank’s guys are awesome!

Make a volume persistent on reboot

Fri, 07 Sep 2012 10:49:00 +0000

Quick and short introduction to a well known bug in nova-volume. I simply want to point it out because I think it could be useful.

Ceph cli output analysis

Fri, 31 Aug 2012 01:55:00 +0000

Brief analysis of the ceph -s command :)

Ceph benchmarks

Sun, 26 Aug 2012 14:23:00 +0000

The time has come to perform some benchmark with Ceph. You expect (or not), they are there, my Ceph’s benchmarks!

Ceph: maintenance mode, use case and common operations

Fri, 17 Aug 2012 12:33:00 +0000

Quick tips about how to manage a production environement. A simple use case here put your ceph journal on a SSD on a production cluster while clients are writting.

Ceph admin socket

Tue, 14 Aug 2012 11:12:00 +0000

The release of the first stable version of Ceph brought a lot of improvment at admin socket level. This is a really useful command that you should use.

Ceph tip: build your monitor map

Thu, 09 Aug 2012 01:21:00 +0000

If you want to change or rebuild a new set of monitor servers, you will need to manipulate the monmap. This monmap is exchange between every monitors.

Tip Ceph: public/private network configuration

Sun, 29 Jul 2012 23:50:00 +0000

While strolling within the ceph mailing list, I came accross something really useful, something that I was looking for :)

Tip: watch your nova log at the same time

Mon, 23 Jul 2012 01:03:00 +0000

I never thought about blogging this, but I think it can be useful for those of you who don’t know this little tool called: Multitail.

Make the network of your VMs fly with the virtio driver

Thu, 19 Jul 2012 00:58:00 +0000

Bring gigabit to your VM’s NIC!

RBD objects analysis

Mon, 16 Jul 2012 16:38:00 +0000

A little bit more about RBD objects…

Delete a VM in an error state

Tue, 10 Jul 2012 17:00:00 +0000

Quick script to delete all the intances in error state.

NFS over RBD

Fri, 06 Jul 2012 23:08:00 +0000

Since CephFS is not most mature component in Ceph, you won’t consider to use it on a production platform. In this article, I offer a possible solution to expose RBD to a shared filesystem.

Remove a MDS server from a Ceph cluster

Wed, 04 Jul 2012 15:15:00 +0000

Quick tip.

If you don’t use CephFS, you don’t need a MDS server. However if you installed it simply to try CephFS and if you’r done playing with it here is how to delete the MDS server.

Download a Glance image from RBD

Tue, 03 Jul 2012 14:41:00 +0000

Quick tip.

If you want to download a glance image stored in a RBD backend.

Use RBD on a client

Sun, 24 Jun 2012 09:42:00 +0000

Quick tip on Ceph. The linux kernel RBD (rados block device) driver allows striping a linux block device over multiple distributed object store data objects. The libceph module takes care of that.

The War for Open Source Clouds

Tue, 08 May 2012 21:26:00 +0000

Little study about the recent release of CloudStack as Apache Licence. Little battle between OpenStack and CloudStack, FIGHT!

Ceph ansible now supports dmcrypt

Mon, 01 Jan 0001 00:00:00 +0000

I recently worked on a new feature that ceph-ansible was laking of: support for dmcrypt. This dmcrypt scenario basically allows you to deploy encrypted OSD data directories. The encrypted key is stored on the monitor’s key/value store. Until recently ceph-ansible wasn’t capable of deploying such configuration. Let’s see how this can be configured.