Team Rubber BlogTeam Rubber talks on the Internet in a blog

We host a number of clients’ websites for them on our own servers. It’s convenient and it saves a lot of deployment headaches, since we know our machines like the backs of our hands.

Yesterday was one of those days.

This chart shows the load average for one of our webservers. It’s – roughly – a measure of how busy the machine is at any given time. As you can see, one of the sites being hosted by that server suddenly started getting hammered around lunchtime yesterday. All very exciting.

If that weren’t excitement enough, the load surfaced a concurrency issue in the site, which had lain dormant until that point. It wasn’t causing data corruption (thankfully – professional ethics would require that I commit seppuku right now if it had), but it was slowing the site down and causing exceptions to get returned in full sight of innocent users.

Cue an immediate dive into the sources to find out what was wrong. By seven o’clock we’d tracked down the issue and designed and tested a viable fix, which both sped the site up and made it possible to spread the load to a second server. Too late for the initial flood, unfortunately, but in plenty of time to help the second wave of visitors which started up this morning.

The final numbers aren’t in yet, but it appears that we’re seeing even more responses today than we did yesterday – and the servers are holding up far more smoothly, with much better response times. It’s stressful when the sky falls in on your head all at once. You feel more alive afterwards, though, for having survived it.

Incidentally, the above graph was generated by Munin, a truly beautiful piece of software. It’s like a heart monitor for servers.