During a restart of the puppet server to add more storage, clients failed to connect to puppetdb. An inspection of the logs showed puppetdb would start then die.
The logs gave the following error message:
EOF Exception caught during broker start, this might be due to KahaDB corruption. Consult the PuppetDB troubleshooting guide.
Along with these:
WARN [main] [broker.BrokerService] Store limit is 100000 mb, whilst the data directory: /var/lib/puppetdb/mq/localhost/KahaDB only has 2497 mb of usable space
2013-07-02 16:26:03,068 ERROR [main] [broker.BrokerService] Temporary Store limit is 50000 mb, whilst the temporary data directory: /var/lib/puppetdb/mq/localhost/tmp_storage only has 2497 mb of usable space
After reading the recommendations here:
http://projects.puppetlabs.com/issues/19239
The KahaDB directory was backed up, then erased and puppetdb restarted successfully.
/var/lib/puppetdb/mq/localhost/KahaDB
At this point, the server was functional again.