Openstack on Sébastien Han

OpenStack and Ceph for Distributed Hyperconverged Edge Deployments

Tue, 12 Mar 2019 16:42:48 +0000

I’m simply relaying an article I reviewed and helped writting. It is reflecting my talk from the last OpenStack Summit in Berlin.

You can read it here Thanks to the author for capturing the essence of the talk.

See you at the OpenStack Summit

Thu, 17 May 2018 16:32:45 +0000

Next week is the OpenStack Summit conference. I will attend and will be giving a talk How to Survive an OpenStack Cloud Meltdown with Ceph.

See you there!

OpenStack Cinder configure replication API with Ceph

Mon, 19 Jun 2017 10:36:06 +0000

I just figured out that there hasn’t been much coverage on that functionality, even though we presented it last year at the OpenStack Summit.

Configure OpenStack Glance for RBD mirroring

Wed, 02 Nov 2016 15:42:17 +0000

Since Ceph Jewel, we have the RBD mirroring functionality and people have been starting using it for multi-site and disaster recovery use cases. The tool is not perfect but is rock solid, expect many enhancements in the future release such as support for multiple peer and daemons. From a pure OpenStack perspective, to enable this feature we don’t really want to add any code into Glance Store. The reason is simple, glance’s store code looks up for specific Ceph features into the Ceph configuration file itself. So there is no point of adding a new configuration flag into Glance that says something like enable_image_journaling. The operator will only have to configure Ceph, that’s it.

Devstack Ceph supports containerized Ceph

Fri, 21 Oct 2016 15:18:20 +0000

Yes people, I’m still alive :). As you might noticed, I’ve been having a hard time to keep up the pace with blogging. It’s mainly due to me traveling a lot these days and preparing conferences. It’s a really busy end of the year for me :).

Fortunately, I’m still finding the time to work on some new features to projects I like. As you might know, I’ve been busy working on ceph-ansible and ceph-docker, trying conciliate both and making sure they work well together. In ceph-docker, we have an interesting container image, that I already presented here. I was recently thinking we could use it to simplify the Ceph bootstrapping process in DevStack. The patch I recently merge doesn’t get ride of the “old” way to bootstrap, the path is just a new addition, a new deployment method.

In practice, this doesn’t change anything for me, but at some point it allows us to validate that a containerized Ceph doesn’t have any problem and bring the same functionality as a non-containerized Ceph. Without further ado, let’s jump into this!

OpenStack Summit Barcelona: time to vote

Mon, 01 Aug 2016 17:12:25 +0000

The next summit will happen in October this year, and it’s already time to vote for your favorite talks!

Quick dive into hyperconverged architecture with OpenStack and Ceph

Mon, 11 Jul 2016 14:16:40 +0000

For those of you who are not familiar with the concept yet, I’m going to give a quick introduction on Hyperconverged infrastructure.

OpenStack Storage for Dummies book

Tue, 05 Jul 2016 15:27:04 +0000

With the help of two colleagues, I’ve been busy writing this little book about storage in OpenStack. The book is quite general but gives some good perspectives on storage challenges you will face in OpenStack. It explains why traditional storage solutions will not work and how Ceph addresses these issues. Ultimately describing why Ceph is the best solution for your OpenStack cloud.

The OpenStack Ceph Galaxy

Mon, 16 May 2016 14:15:02 +0000

Picture of our galaxy :). This picture describes the state of the integration of Ceph into OpenStack.

OpenStack Cinder: discard support for Ceph in Mitaka

Wed, 11 May 2016 14:49:20 +0000

OpenStack Mitaka brought the support of a new feature. This feature is a follow-up of the Nova discard implementation. Now it is possible to configure Cinder per backend.

OpenStack Summit Austin: Time To Vote

Wed, 10 Feb 2016 11:06:00 +0000

The summit is almost there and it is time to vote for the presentation you want to see :).

OpenStack Nova snapshots on Ceph RBD

Mon, 05 Oct 2015 17:10:00 +0000

I have been waiting for this feature for more than a year and it is almost there! This likely brings us one step toward diskless compute nodes. This “under the hood” article will explain the mechanisms in place to perform fast and efficient Nova instance snapshots directly in Ceph.

OpenStack Nova: configure multiple Ceph backends on one hypervisor

Tue, 15 Sep 2015 11:11:00 +0000

Configure a Nova hypervisor with more than one backend to store the instance’s root ephemeral disks.

Ceph at the OpenStack Summit Tokyo 2015

Fri, 04 Sep 2015 11:17:00 +0000

With this article, I would like to take the opportunity to thank you all for voting for our presentation. It is always with a great pleasure that we will give you the last updates of Ceph developments happening in OpenStack.

Ceph talks coverage at the next OpenStack summit:

OpenStack Summit Tokyo: time to vote

Fri, 24 Jul 2015 10:29:00 +0000

Yet again and for the second time this year, it is time to vote for summit presentations :). Self promotion ahead :).

As always, Josh and I will present the newest addition of Liberty for Ceph in OpenStack. I don’t want to spoil to much but what I can tell you is this cycle is doing well and most of the wanted features would likely land in Liberty. So if you want to see all the amazing things that happened during this cycle:

Ceph and OpenStack: current integration and roadmap

This presentation will be a follow up on Dude where’s my volume? talk from Vancouver.

State of Multi-Site Storage in OpenStack

Thanks in advance for your votes and support :).

OpenStack cinder: readonly device

Mon, 08 Jun 2015 00:01:00 +0000

It is possible to attach readonly volume to a virtual machine.

OpenStack Glance: use multiple location for an image

Wed, 13 May 2015 00:01:00 +0000

This feature is pretty old (introduced in the Havana cycle if I remember correctly), I just never got the chance to play with it. However I believe this feature could benefit to many users. Let’s see how we can get the best of it.

OpenStack Glance: a first glimpse at image conversion

Mon, 11 May 2015 01:23:00 +0000

Following my best Kilo’s additions selection, today I will be introducing the Glance image conversion. This feature was discussed at the last OpenStack summit in Paris, you can access the etherpad discussion. Before you get all excited, let me tell you first that the patch introduced during this Kilo cycle is the first of a series. So do not get disappointed if it does not fit your needs yet (and it probably won’t…). Now if you are still inclined reading the article let’s jump in!

OpenStack Glance: deactivate an image

Wed, 06 May 2015 00:12:00 +0000

Kilo has been released last week. This blog post is the first of a series that will demonstrate some nifty new features.

Managing cloud images life cycle is a real pain for public cloud providers. Since users have the ability to import their own images they can potentially introduce vulnerabilities with them. Thus the cloud operators should be able to deactivate (temporary) an image to inspect it. Later operators can reactivate it or just remove it if they believe the image is a threat for the cloud environment.

Another use case, as well is for cloud image updates, while performing the update of an image the operator might want to hide it from all the users. Then when the update is complete he can reactivate the image so the users can boot virtual machines from it.

OpenStack Summit Vancouver: thanks for your votes

Sun, 29 Mar 2015 23:33:00 +0000

Bonjour, bonjour ! Quick post to let you know that my talk submission has been accepted, so I’d like to thank you all for voting. As a reminder, our talk (Josh Durgin and I) is scheduled Tuesday, May 19 between 11:15am - 11:55am.

Also note that the summit has other Ceph talks!

See you in Vancouver!

OpenStack: reserve memory on your hypervisors

Mon, 23 Mar 2015 15:09:00 +0000

One major use case for operators is to be able to reserve a certain amount of memory in the hypervisor. This is extremely useful when you have to recover from failures. Imagine that you run all your virtual machines on shared storage (Ceph RBD or Sheepdog or NFS). The major benefit from running your instances on shared storage is that it will ease live-migration and evacuation. However, if a compute node dies you want to make sure that you have enough capacity on the other compute nodes to relaunch your instances. Given that the nova host-evacuate call goes through the scheduler again you should get an even distribution.

But how to make sure that you have enough memory on the other hypervisors? Unfortunately there is no real memory restriction mechanism. In this article I will explain how we can mimic such behavior.

OpenStack Glance NFS and Compute local direct fetch

Mon, 16 Mar 2015 16:12:00 +0000

This feature has been around for quite a while now, if I remember correctly it was introduced in the Grizzly release. However, I never really got the chance to play around with it. Let’s assume that you use NFS to store Glance images, we know that the default booting mechanism implies to fetch the instance image from Glance to the Nova compute. This is basically streaming the image which involves network throughput and makes the boot process longer. OpenStack Nova can be configured to directly access Glance images from a local filesystem path. This is ideal for our NFS scenario.

OpenStack guest and watchdog

Mon, 09 Mar 2015 12:04:00 +0000

Libvirt has the ability to configure a watchdog device for QEMU guests. When the guest operating system hangs or crashes the watchdog device is used to automatically trigger some actions. The watchdog support was added in OpenStack Icehouse.

OpenStack and Backup

Tue, 17 Feb 2015 12:14:00 +0000

Doing backups in the Cloud is not an easy task. In this article I will try to answer some frequently asked questions.

OpenStack summit Vancouver talks: Ceph and OpenStack upgrades

Tue, 17 Feb 2015 00:52:00 +0000

Self promotion ahead :) For the next OpenStack summit I have submitted two talks.

OpenStack: perform consistent snapshots with Qemu Guest Agent

Mon, 09 Feb 2015 16:17:00 +0000

A while back, I wrote an article about taking consistent snapshots of your virtual machines in your OpenStack environment. However this method was really intrusive since it required to be inside the virtual machine and to manually summon a filesystem freeze. In this article, I will use a different approach to achieve the same goal without the need to be inside the virtual machine. The only requirement is to have a virtual machine running the qemu-guest-agent.

OpenStack and Ceph: RBD discard

Mon, 02 Feb 2015 09:32:00 +0000

Only Magic Card player might recognize that post picture :) (if you’re interested)

I have been waiting for this for quite a while now. Discard, also called trim (with SSD), is a space reclamation mechanism that allows you to reclaim unused blocks on a disk. RBD images are sparse by default, this means that the space they occupy increase the more you write data (opposite of preallocation). So while writing on your filesystem you might end up to the end of your device. On the Ceph side, no one knows what is happening on the filesystem, so we actually end up with fully allocated blocks… In the end the cluster believes that the RBD images are fully allocated. From an operator perspective, having the ability to reclaim back the space unused by your running instances is really handy.

OpenStack Cinder with Ceph under the hood

Fri, 23 Jan 2015 11:21:00 +0000

What’s happening under the hood while playing with Cinder and Ceph? Answer table :-).

OpenStack: disable a compute node during its first bootstrap

Mon, 19 Jan 2015 05:48:00 +0000

For operationnal reasons, you might not want to automatically make your compute node available. With the following flag, during its first bootstrap the compute node will register itself to the service list. However it will be disabled, so virtual machines can not be scheduled on it:

enable_new_services=False

OpenStack configure VM migrate nova SSH

Tue, 06 Jan 2015 13:46:00 +0000

Just a reminder for me :D.

OpenStack: import existing Ceph volumes in Cinder

Tue, 09 Dec 2014 16:14:00 +0000

This method can be useful while migrating from one OpenStack to another.

OpenStack Glance: import images and convert them directly in Ceph

Tue, 11 Nov 2014 23:45:00 +0000

Ceph, to work in optimal circumstances requires the usage of RAW images. However, it is painful to upload RAW images in Glance because it takes a while. Let see how we can make our life easier.

OpenStack Glance: disable cache management while using Ceph RBD

Mon, 03 Nov 2014 11:00:00 +0000

The OpenStack documentation often recommends to enable the Glance cache while using the default store file, with the Ceph RBD backend things are slightly different.

OpenStack Glance: allow user to create public images

Thu, 30 Oct 2014 10:38:00 +0000

Since Juno, it is not possible anymore for an user to create public images nor make one of his images/snapshots public. Even though this new Glance policy is a good initiative, let’s see how we can get the old behavior back.

Interested in Ceph? Join us at the OpenStack summit in Paris!

Sun, 26 Oct 2014 13:46:00 +0000

The next OpenStack summit is just around the corner and as usual Josh Durgin and I will lead the Ceph and OpenStack design session. This session is scheduled for November 3 from 11:40 to 13:10, find the description link here. The etherpad is already available here so don’t hesitate to add your name to the list along with your main subject of interest. See you in Paris!

OpenStack at the CephDays Paris

Fri, 05 Sep 2014 10:25:00 +0000

Save the date (September 18, 2014) and join us at the new edition of the Ceph Days in Paris. I will be talking about the new amazing stuff that happened during this (non-finished yet) Juno cycle. Actually I’ve never seen so many patch sets in one cycle :D. Things are doing well for Ceph in OpenStack! Deploying Ceph with Ansible will be part of the talk as well.

The full schedule is available, don’t forget to register to the event.

Hope to see you there!

OpenStack: use ephemeral and persistent root storage for different hypervisors

Mon, 01 Sep 2014 14:41:00 +0000

Computes with Ceph image backend and computes with local image backend. At some point, you might want to build hypervisor and use their local storage for virtual machine root disks. Using local storage will help you maximasing your IOs and will reduce IO latentcies to the minimum (compare to network block storage). However you will lose handy features like the live-migration (block migration is still an option but slower). Data on the hypervisors will not have a good availability level too. If the compute node crashes the user will not be able to access his virtual machines for a certain amount of time. On another hand, you want to build hypervisors that where virtual machine root disks will live into Ceph. Then you will be able to seemlessly move virtual machine with the live-migration. Virtual machine disks will be highly available so if a compute node crashes you can quickly evacuate the virtual machine disk to another compute node. Ultimately, your goal is to dissociate them, fortunately for you OpenStack provides a mechanism based on host agregate that will help you achieve your objective. Thanks agregate filters you will be able to expose these hypervisors.

A must-have cron job on your OpenStack Cloud

Mon, 18 Aug 2014 11:46:00 +0000

Running OpenStack on production can be difficult, so every optimizations are good to take :).

Start considering Ceph as a backend for OpenStack Cinder (to replace LVM)

Mon, 02 Jun 2014 19:23:00 +0000

Just back from the Juno summit, I attended most of the storage sessions and was extremely shocked how Ceph was avoided by storage vendors. However LVM, the reference storage backend for Cinder was always mentioned. Maybe, is it a sign that Ceph is taking over? Talking about LVM, the last OpenStack survey showed that it was the more used backend.

Back from the Juno summit Ceph integration into OpenStack

Thu, 29 May 2014 16:32:00 +0000

Six months have passed since Hong Kong and it is always really exciting to see all the folks from the community gathered all-together in a (bit chilly) convention center. As far I saw from the submitted and accepted talks, Ceph continues its road to the top. There is still a huge growing interest about Ceph. On Tuesday May 13th, Josh and I led a (3 hours long) session to discuss the next steps of the integration of Ceph into OpenStack. To be honest, back when we were in Hong Kong, I believe that we were too optimistic about our roadmap. So this time we decided to be a little more realistic and took a more step-by-step approach rather than “let’s add everything we can”. However, this does not mean that the Icehouse cycle was limited in terms of features, not at all! Indeed the Icehouse cycle has seen some tremendous improvements. I don’t know if you remember my last year article right after the Icehouse summit, there was a feature that I wanted so much: RADOS as a backend for Swift. And yes, we made it, so if you want more details you’d better continue the reading :).

Don't burn down your OpenStack cloud

Mon, 31 Mar 2014 00:01:00 +0000

Managing an OpenStack public cloud can be tough and building it properly is even harder. You can not predict the workload of your platform, customers do what they want (yes they pay for this!). So yes, cloud performance are often unpredictable! Recent studies showed that while running a long-standing benchmark on several cloud platforms, they experienced a performance drop-down of 40% (crazy isn’t it?). However, there are some simple facilities in OpenStack that allow you to have a better control of the resources that you offer to your customers/users. This is what I am going to briefly explore in this article.

OpenStack, Ceph RBD and QoS

Mon, 23 Dec 2013 00:19:00 +0000

The Havana cycle introduced a QoS feature on both Cinder and Nova. Quick tour of this excellent implementation.

Back from Icehouse OpenStack summit: Ceph/OpenStack integration

Tue, 26 Nov 2013 16:55:00 +0000

The summit was exciting and full of good things and announcements. We had great Cinder sessions and an amazing Ceph/OpenStack integration session. I’ve led the Ceph/OpenStack integration session with Josh Durgin (Inktank). We had a good participation from the audience. I would like to specially thank Sage Weil, Haomai Wang, Edward Hope-Morley for their good inputs. The main purpose of this session was to gather ideas to improve the Ceph integration into Openstack. Eventually, we built a draft of the Icehouse’s roadmap. For those of you who were not attending the session and are curious to learn more about what’s going to happen in the next few months this article is for you!

OpenStack survey 2013

Wed, 06 Nov 2013 08:56:00 +0000

Ceph 17% ! :D

Build a PaaS zone within your OpenStack cloud

Thu, 31 Oct 2013 11:07:00 +0000

OpenStack has seen a lot of PaaS oriented project coming for the last few months. Heat, the orchestration service was introduced during Grizzly, Havana just got the support of the Docker hypervisor and Trove the Database as a Service project is planned for Icehouse. More recently, Manila the Distributed Filesystem as a Service and Raksha the Data Protection As Service appeared.

So, want to bring OpenStack to the next level?

CoreOS, a good operating system for your OpenStack controllers

Thu, 10 Oct 2013 11:33:00 +0000

During this article, I am going to explain why I believe that CoreOS is an excellent operation system for your OpenStack controllers.

OpenStack Heat and Ceilometer got their dashboard panel

Fri, 06 Sep 2013 22:54:00 +0000

The Havana milestone release of the Horizon dashboard brought an absolutely wonderful panel for Heat, the orchestration service and Ceilometer, the metering service. Quick preview before the Havana’s official release.

OpenStack Havana flush token manually

Fri, 06 Sep 2013 15:05:00 +0000

It has always been a huge pain to manage token in MySQL espacially with PKI token since they are larger than UUID token. Almost a year ago I wrote an article to purge token via a script. So finally, we have an easy option to purge all expired token.

Openstack: unexplained high CPU load on compute nodes

Mon, 12 Aug 2013 10:26:00 +0000

Recently I encountered a weird issue: the CPU load of my hosts was quite high with a load average of 20 and picks to 40. First, I had a look at the guest system since the high load was generated by guests. Unfortunately the VM was not doing anything, no IOPS and no CPU.

OpenStack 3rd years retrospective

Mon, 22 Jul 2013 11:45:00 +0000

OpenStack: instance evacuation goes to host

Fri, 19 Jul 2013 16:02:00 +0000

With Grizzly came the instance evacuation but a quite recent addition to the code expended it to hosts.

OpenStack: perform a live migration using a specific NIC

Tue, 02 Jul 2013 17:58:00 +0000

Quick old tip (back from the cave) to trigger a live migration from a specific network card.

What I think about CephFS in OpenStack

Mon, 24 Jun 2013 00:28:00 +0000

I recently had some really interesting questions that led to some nice discussions. Since I received the same question twice, I thought it might be good to share the matter with the community.

The question was pretty simple and obvioulsy the context is about OpenStack:

See you at the OpenStack summit

Thu, 11 Apr 2013 09:43:00 +0000

Next week is the OpenStack Summit conference. eNovance’s team will be present at the event. The convention is going to be awesome, I saw plenty of amazing sessions and I really look forward to attending those talks. Speaking about session, Emilien Macchi and I will give a talk about High Availability in OpenStack. You can check the talk page here. Hope to see you there!

Openstack: quickly fix mirrored queues errors

Tue, 09 Apr 2013 22:10:00 +0000

Just started with Grizzly and already been through some minor issues :).

OpenStack: override DHCP information sent by DNSMASQ to a VM

Mon, 18 Feb 2013 11:21:00 +0000

One of the most annoying thing is when the resolv.conf of your VM keeps changing because of the information sent by the DNSMASQ process. In this article, I assume that your setup has some conventions such as:

One network per customer, with fixed_ips range with something like 10.100.$ID.0/24
One ID (number) per tenant

OpenStack Nova and availability zones

Thu, 24 Jan 2013 17:22:00 +0000

Availability zone in OpenStack. The main purpose of this article is to play a bit with availability zones.

OpenStack maintenance mode

Wed, 09 Jan 2013 18:02:00 +0000

Some tips for performing smooth maintenance in OpenStack.

OpenStack: perform consistent snapshots

Mon, 10 Dec 2012 10:30:00 +0000

Make consistent instance or volume snapshots.

Openstack and rsyslog

Wed, 05 Dec 2012 17:12:00 +0000

I don’t want to see my logs poluted by INFO messages saying “hey, I’m running!”. No thanks, it burns my I/O cycles for nothing. If it doesn’t work I have a monitoring system for that and if it’s broken I simply change the log level to DEBUG. Of course it’s personal thoughts, however for those you who are interested just click ;-).

Tip: OpenStack Retrieve usage statictics

Fri, 09 Nov 2012 23:45:00 +0000

While waiting for the Ceilometer project to be ready, nova provides some facilities to retrieve simple statistics. A little overview about the available commands.

Openstack: memory overcommit

Sat, 22 Sep 2012 19:06:00 +0000

I couple of days ago I noticed that I never tested the overcommit capability in OpenStack

Openstack: play with quota

Wed, 19 Sep 2012 22:01:00 +0000

Short introduction to quota in OpenStack.

OpenStack: Auto assign floating IP

Fri, 03 Aug 2012 23:02:00 +0000

It can be really handy to auto assign floating IP addresses to every new instance.

OpenStack block migration

Thu, 12 Jul 2012 15:30:00 +0000

Block migration, the best compromise possible?

Play with OpenStack instance metadata

Wed, 11 Jul 2012 22:39:00 +0000

Nova API metadata.

OpenStack: Nova components HA

Mon, 02 Jul 2012 00:25:00 +0000

I wrote all the missing resource agents related to the nova ecosystem. All the RAs are available on my Github. All the ’nova’ RAs mainly re-use the structure of the resource agent written by Martin Gerhard Loschwitz from Hastexo. See here.

OpenStack: Glance and Keystone HA

Thu, 28 Jun 2012 13:02:00 +0000

The purpose of this article is to achieve high-availability for some OpenStack components.

Setup Cloud Pipe VPN in OpenStack

Wed, 20 Jun 2012 11:39:00 +0000

This article mainly re-uses the OpenStack official documentation. Since the latter has errors in it, I fixed them. It’s fully functionnal under Ubuntu 12.04 distro.

Delete a tenant in OpenStack

Thu, 14 Jun 2012 17:41:00 +0000

If you are running on OpenStack Essex, you should have some problem to delete a tenant.

Introducing Ceph to OpenStack

Sun, 10 Jun 2012 00:04:00 +0000

OpenStack High Availability: RabbitMQ

Mon, 21 May 2012 11:28:00 +0000

Rabbitmq has his own buildin cluster management system. Here, we don’t need Pacemaker, everything is managed by RabbitMQ itself.

OpenStack High Availability 1/??

Thu, 17 May 2012 13:37:00 +0000

First article of a long serie to build an highly available OpenStack platform. This one is more a state of art about the OpenStack HA.