Sometime last night, mysql decided that it would no longer resolve incoming hosts. This wasn’t a problem for the existing open connections, so didn’t effect the hab.la service, but it did affect new connections from the webservers, and if there is one thing rails doesn’t really complain about that much in the log files, is a mysql connection that takes forever to establish.
I am not sure of the exact cause of the error, I have encountered it before, and it is really easy to diagnose. Here’s what it looks like:
| 3436942 | unauthenticated user | 192.168.0.52:49607 | | Connect | | login | | | 3436943 | unauthenticated user | 192.168.0.52:49608 | | Connect | | login | | | 3436944 | unauthenticated user | 192.168.0.52:49609 | | Connect | | login | | | 3436945 | unauthenticated user | 192.168.0.52:49610 | | Connect | | login | | | 3436946 | unauthenticated user | 192.168.0.52:49611 | | Connect | | login | | | 3436947 | unauthenticated user | 192.168.0.52:49612 | | Connect | | login | | | 3436948 | unauthenticated user | 192.168.0.2:41049 | | Connect | | login | |
You can read more about how to solve this error at ”http://bugs.mysql.com/bug.php?id=8945”
Basically to fix this error add
skip-name-resolv to your /etc/mysql.cnf. And cleanly restart your mysql-server.
Of course to cleanly restart your mysql server you will make sure that any reconnecting service that you might have stops reconnecting long enough for mysql to complete the shutdown process. (And for Hab.la this means to do a clean restart we need to shut down a lot of of services) you can read more about the mysql shutdown procedure here: http://dev.mysql.com/doc/refman/5.0/en/server-shutdown.html