SimpleGeo Blog

Recently at SimpleGeo we had some issues with our RabbitMQ setup. This wasn’t really an issue with RabbitMQ per se, but an issue with how we’d laid out the architecture of it. It was brought up that maybe we should rewrite our consumers in a different language, which, of course, lead to a huge debate about which language we should switch to.

I’ve said this many times before, but languages don’t scale. Fundamentally speaking, our issues with RabbitMQ came down to architecture and, as it turns out, our old foe I/O. So what were our issues?

• First off, we were running RabbitMQ on small EC2 instances. It’s a known bug that RabbitMQ crashes when it runs out of RAM. So we moved our RabbitMQ boxes to larger instances with more RAM.
• We had serious CPU contention issues that ended up eating up a ton of RAM. This was because we were running our consumers on the queue instances themselves. We’ve since moved the consumers to high CPU instance types.
• The reason our consumers were taking so long to munch through a large queue was because we had monolithic job code. Turns out that the I/O overhead of S3 backups and URL callbacks, which are entirely network I/O bound, were holding up the faster portions of the jobs. We split our consumers into many small jobs and things are working much better now.

So which language did we decide on? We didn’t really. We’ve already deployed code into production using Python, Java, and PHP. I think we’re even running a Ruby proxy of some sort. We’re seriously looking at Erlang and Scala as well. As the manager at SimpleGeo, I really only have two rules when it comes to picking languages:

1. It must have a demonstrated record of working well in high-load/high-volume production environments. Java, Scala, Erlang, PHP, and Python all fall into this category. Haskell not so much.
2. I must have two coders on staff who can code in that given language or are interested in coding in that language. I call this The Bus Rule because I need someone around in case one of them gets hit by a bus.

That’s it. Choose something that’s demonstrably effective, which has buy-in and mindshare in the office and run with it.