Integrated Backup Plan

So... last night couple of our application servers crashed. It was apparently due to some new codes that were released although I'm not yet sure. Here is the summarize version of what transpired:

1. Sysadmin got paged that the application servers were not responding.

2. Sysadmin called the DBA team to check the database connections... We said everything looks okay on our end.

3. Sysadmin called the developers next and they found out that the new codes caused the system to go haywire.

4. When the developers tried to revert back the old codes, apparently some of them did not checked-in the older version and it caused a couple missing database privileges to occur. Also, some needed tables were also missing.

5. From that time on, we were really in trouble and sufficient to say it was a very very long night.

The lesson: maybe it is nice to test an integrated backup plan once in awhile. We always think of doing database backups but it is just one component of the whole system. Anyway, that was what I was thinking during the time and the number of hours I have lost because of the incident.

MySQL SandBox Gem

Consider these scenarios:

1. You need to test some new configurations in your my.cnf and you need to reboot MySQL.
2. You want to run some benchmarking on your database.
3. You want to try some new replication ideas that involves creating new replication structure, stopping the slaves, doing failovers, or any ideas related to replication.

And, most importantly, you want to do all of the above without disturbing your production databases.

Answer:

MySQL Sandbox. A lot of people have written about this gem but until you have use it, you will not realize how simple and yet valuable this tool is.