The problem with Hab.la is there are too many distinct services running. In this case the server that handles routing connections to MSN and AOL was essentially multiplexing the log files, because there was still some debugging code kicking around in the codebase that was having the affect of opening multiple log files, and sending extra data to these log files. – this also made tracing the log files very strange because you would have the same entry multiple times.
And I was wondering why things were being sent to stdout in duplicate, ha.
In addition to just writing much more data than it needed to write, it was also logging to directory without a log rotation facility on a relatively small partition (i.e. /tmp)
Needless to say, when / fills up, a lot of things go crazy.
I would just add a log rotate rule to /tmp, but I am not sure how well logrotate places with python’s logging facility.
(Looks like we should start using WatchedFileHandler if we are going to play nice with log rotate – http://docs.python.org/library/logging.html, that seems like a high priority todo )