OpenStack Summit Portland: High Availability from DevOps Side
Date: 18/04/2013. Video: Slides: High Availability from the DevOps side - OpenStack Summit Portland from eNovance
Some Ceph experiments
Sometimes it’s just funny to experiment the theory, just to notice “oh well it works as expected”. This is why today I’d like to share some experiments with 2 really specific flags: noout and nodown. Behaviors describe in the article are well known because of the design of Ceph, so don’t yell at me: ‘Tell us something we don’t know!’, simply see this article a set of exercises that demonstrate some Ceph internal functions :-).
See you at the OpenStack summit
Next week is the OpenStack Summit conference. eNovance’s team will be present at the event. The convention is going to be awesome, I saw plenty of amazing sessions and I really look forward to attending those talks. Speaking about session, Emilien Macchi and I will give a talk about High Availability in OpenStack. You can check the talk page here. Hope to see you there!
Openstack: quickly fix mirrored queues errors
Just started with Grizzly and already been through some minor issues :).
Ceph Puppet Modules
Quite recently François Charlier and I worked together on the Puppet modules for Ceph on behalf of our employer eNovance. In fact, François started to work on them last summer, back then he achieved the Monitor manifests. So basically, we worked on the OSD manifest. Modules are in pretty good shape thus we thought it was important to communicate to the community. That’s enough talk, let’s dive into these modules and explain what do they do. See below what’s available:
- Testing environment is Vagrant ready.
- Bobtail Debian latest stable version will be installed
- The module only supports CephX, at least for now
- Generic deployment for 3 monitors based on a template file examples/common.sh which respectively includes mon.sh, osd.sh, mds.sh.
- Generic deployment for N OSDs. OSD disks need to be set from the examples/site.pp file (line 71). Puppet will format specified disks in XFS (only filesystem implemented) using these options:
-f -d agcount=<cpu-core-number> -l size=1024m -n size=64kand finally mounted with:rw,noatime,inode64. Then it will mount all of them and append the appropriate lines in the fstab file of each storage node. Finally the OSDs will be added into Ceph.
All the necessary materials (sources and how-to) are publicly available (and for free) under AGPL license on eNovance’s Github. Those manifests do the job quite nicely, although we still need to work on MDS (90% done, just need a validation), RGW (0% done) and a more flexible implementation (authentication and filesystem support). Obviously comments, constructive critics and feedback are more then welcome. Thus don’t hesitate to drop an email to either François ([email protected]) or I ([email protected]) if you have further questions.
Grizzly availability zones
Short short update.
Ceph: change PG number on the fly
A Placement Group (PG) aggregates a series of objects into a group, and maps the group to a series of OSDs. A common mistake while creating a pool is to use the rados command which by default creates a pool of 8 PGs. Sometime you don’t properly know how to set this value thus you use the ceph command but put an extremely high value for it. Both case are bad and could lead to some unfortunate situations. In this article, I will explore some methods to work around this major problem.
MySQL and general logs
Update the general_log variable while MySQL runs.
OpenStack: override DHCP information sent by DNSMASQ to a VM
One of the most annoying thing is when the resolv.conf of your VM keeps changing because of the information sent by the DNSMASQ process. In this article, I assume that your setup has some conventions such as:
- One network per customer, with fixed_ips range with something like
10.100.$ID.0/24 - One ID (number) per tenant
Purge some MySQL binary logs
The default value of the variable expire_logs_days is 10 days, most of the time this is way too long. This mini how-to shows how to change this value. Fortunately expire_logs_days is a dynamic variable so we can edit it while MySQl runs, we don’t need to restart the server.