Sébastien Han

| Comments | Tag: ceph

Cleanup keystone tokens

Every once in a while you really want to cleanup the token table of the Keystone database. A couple of weeks ago while backuping my cloud controller I noticed that the backup of the Keystone database was longer than the other databases. After that, I checked the size of the dump (compressed) 60MB. Hummm but there is almost nothing in the Keystone database: users, tenants… wait.. could it be TOKENS?!

The token validity is manage via the following options in keystone.conf:

[token]
driver = keystone.token.backends.sql.Token

# Amount of time a token should remain valid (in seconds)
expiration = 86400

One option could be to use different backend to store the tokens:

  • The keystone.token.backends.memcache, Memcached storage backend
  • The keystone.token.backends.kvs, Key Value storage backend

I will prefer another backend to store the tokens in order to make database dump shorter and smaller. I’m not quite sure if memcache is a good candidat though. This could make things harder for some reasons like:

  • Does the token remain forever in memcache?
  • Cache consistency, if a server crash
  • Makes the setup more complex, try to achieve a replicated memcache

See this launchpad discussion for more details.

Every nova/glance/cinder commands ask for a token while trying to execute a command.

I personnally end up with the following:

$ sudo mysql -uroot -p -e 'USE keystone; SELECT * FROM token;' | wc -l
Enter password: 
1970938

The setup runs for 2 months now and already 1970938 and I don’t run a public cloud. I can’t imagine the nightmare with a public cloud…

Little bash script:

#!/bin/bash

mysql_user=
mysql_password=
mysql_host=

mysql -u${mysql_user} -p${mysql_password} -h${mysql_host} -e 'USE keystone ; DELETE FROM token WHERE NOT DATE_SUB(CURDATE(),INTERVAL 2 DAY) <= expires;'

If during the process you endup with this error:

ERROR 1205 (HY000) at line 1: Lock wait timeout exceeded; try restarting transaction

Simply increase the innodb_lock_wait_timeout:

mysql> show variables like 'innodb_lock_wait_timeout';
+--------------------------+-------+
| Variable_name            | Value |
+--------------------------+-------+
| innodb_lock_wait_timeout | 50    |
+--------------------------+-------+
1 row in set (0.00 sec)

mysql> set innodb_lock_wait_timeout=100;
Query OK, 0 rows affected (0.00 sec)

mysql> show variables like 'innodb_lock_wait_timeout';
+--------------------------+-------+
| Variable_name            | Value |
+--------------------------+-------+
| innodb_lock_wait_timeout | 100   |
+--------------------------+-------+

Then re-run your command again.

Note: I volontary let a retention of 2 days in the command since I work with days and not with hours. It’s not always day per day, thus some token could overlap and be valid. So 2 days are fine.


II. Bonus

II.1. Retrieve a token with curl

If you want to retrieve a token via curl:

$ curl -s -d "{\"auth\":{\"passwordCredentials\": {\"username\": \"admin\", \"password\": \"admin\"}, \"tenantName\": \"admin\"}}" -H "Content-type: application/json" http://127.0.0.1:5000/v2.0/tokens | tr ',' 'n' | grep '"id":' | cut -d'"' -f12 | head --lines 1
ff2d33b0de69431d9203a5fa43efa903

II.2. Cron task for the cleanup script

Script to execute periodically:

#!/bin/bash

# Purpose of the script
# Everytime a service wants to be do 'something' it has to retrieve an autentication token
# Nova/Glance/Cinder services are manage by Pacemaker and monitor functions (from the RA) ask for a token every 10 sec
# There is no cleanup procedure nor periodical task running to delete expire token

mysql_user=keystone
mysql_password=********
mysql_host=
mysql=$(which mysql)

logger -t keystone-cleaner "Starting Keystone 'token' table cleanup"

logger -t keystone-cleaner "Starting token cleanup"
mysql -u${mysql_user} -p${mysql_password} -h${mysql_host} -e 'USE keystone ; DELETE FROM token WHERE NOT DATE_SUB(CURDATE(),INTERVAL 2 DAY) <= expires;'
valid_token=$($mysql -u${mysql_user} -p${mysql_password} -h${mysql_host} -e 'USE keystone ; SELECT * FROM token;' | wc -l)
logger -t keystone-cleaner "Finishing token cleanup, there is still $valid_token valid tokens..."

exit 0

II.3. Use memcached to store Keystone tokens

Install memcached:

$ sudo apt-get install memcached -y

Put the following option in your keystone.conf:

[token]
driver = keystone.token.backends.memcache.Token

Then restart the service Keystone:

$ sudo service keystone restart

Check if the connection is well established:

$ sudo lsof -i :11211
COMMAND     PID     USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
memcached  2900 memcache   26u  IPv4 20295625      0t0  TCP localhost:11211 (LISTEN)
memcached  2900 memcache   27u  IPv4 20295626      0t0  UDP localhost:11211 
memcached  2900 memcache   29u  IPv4 20307867      0t0  TCP localhost:11211->localhost:37623 (ESTABLISHED)
memcached  2900 memcache   30u  IPv4 20307869      0t0  TCP localhost:11211->localhost:37626 (ESTABLISHED)
memcached  2900 memcache   31u  IPv4 20307925      0t0  TCP localhost:11211->localhost:37634 (ESTABLISHED)
keystone- 10599 keystone   10u  IPv4 20308147      0t0  TCP localhost:37623->localhost:11211 (ESTABLISHED)
keystone- 10599 keystone   13u  IPv4 20308153      0t0  TCP localhost:37626->localhost:11211 (ESTABLISHED)
keystone- 10599 keystone   15u  IPv4 20308221      0t0  TCP localhost:37634->localhost:11211 (ESTABLISHED)

Retrieve cache object count:

$ sudo telnet 127.0.0.1 11211
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
stats items
STAT items:9:number 4
STAT items:9:age 314
STAT items:9:evicted 0
STAT items:9:evicted_nonzero 0
STAT items:9:evicted_time 0
STAT items:9:outofmemory 0
STAT items:9:tailrepairs 0
STAT items:9:reclaimed 0
STAT items:9:expired_unfetched 0
STAT items:9:evicted_unfetched 0
STAT items:10:number 1310
STAT items:10:age 511
STAT items:10:evicted 0
STAT items:10:evicted_nonzero 0
STAT items:10:evicted_time 0
STAT items:10:outofmemory 0
STAT items:10:tailrepairs 0
STAT items:10:reclaimed 0
STAT items:10:expired_unfetched 0
STAT items:10:evicted_unfetched 0
END
quit
Connection closed by foreign host.

You should see the STAT items:10:number growing and growing.

Simply run the script aaaaaannnd it’s gone!

Comments