Sometimes removing OSD, if not done properly can result in double rebalancing. The best practice to remove an OSD involves changing the crush weight to 0.0 as first step.

So in the end, this will give you:

$ ceph osd crush reweight osd.<ID> 0.0

Then you wait for rebalance to be completed. Eventually completely remove the OSD:

$ ceph osd out <ID>
$ service ceph stop osd.<ID>
$ ceph osd crush remove osd.<ID>
$ ceph auth del osd.<ID>
$ ceph osd rm <ID>

Et voilà !