Recently at SimpleGeo we had some issues with our RabbitMQ setup. This wasn’t really an issue with RabbitMQ per se, but an issue with how we’d laid out the architecture of it. It was brought up that maybe we should rewrite our consumers in a different language, which, of course, lead to a huge debate about which language we should switch to.

I’ve said this many times before, but languages don’t scale. Fundamentally speaking, our issues with RabbitMQ came down to architecture and, as it turns out, our old foe I/O. So what were our issues?

• First off, we were running RabbitMQ on small EC2 instances. It’s a known bug that RabbitMQ crashes when it runs out of RAM. So we moved our RabbitMQ boxes to larger instances with more RAM.
• We had serious CPU contention issues that ended up eating up a ton of RAM. This was because we were running our consumers on the queue instances themselves. We’ve since moved the consumers to high CPU instance types.
• The reason our consumers were taking so long to munch through a large queue was because we had monolithic job code. Turns out that the I/O overhead of S3 backups and URL callbacks, which are entirely network I/O bound, were holding up the faster portions of the jobs. We split our consumers into many small jobs and things are working much better now.

So which language did we decide on? We didn’t really. We’ve already deployed code into production using Python, Java, and PHP. I think we’re even running a Ruby proxy of some sort. We’re seriously looking at Erlang and Scala as well. As the manager at SimpleGeo, I really only have two rules when it comes to picking languages:

1. It must have a demonstrated record of working well in high-load/high-volume production environments. Java, Scala, Erlang, PHP, and Python all fall into this category. Haskell not so much.
2. I must have two coders on staff who can code in that given language or are interested in coding in that language. I call this The Bus Rule because I need someone around in case one of them gets hit by a bus.

That’s it. Choose something that’s demonstrably effective, which has buy-in and mindshare in the office and run with it.

Over the last few months we’ve been fortunate to be able to hire four amazing engineers, so we’re excited to announce the two latest additions to our team, Joel Longtine and Wade Simmons, do not upset the trend. They will be joining the SimpleGeo family on February 1st.

Joel and Wade join us from AOL where they worked on AIM Lifestream, which grew out of AOL’s acquisition of my last company, Socialthing. When Joe and I found out they were leaving AOL at the end of 2009 to pursue new opportunities we jumped at the chance to hire a proven team of engineers who enjoyed working together and who had proven themselves scaling one of AOL’s newest products.

They’ll be responsible for building out an exciting new product that we’re not quite ready to announce. It will give Wade and Joel plenty of opportunity to leverage their backgrounds in scaling real-time environments. We’re eagerly anticipating the day we can further describe this new product.

And then there were nine. We’re excited that we’ve managed to hire so many amazing engineers so quickly and expect to be announcing more exciting additions to our team over the next few months.

Last fall, Joe and I had met with Dan Shoutis to talk about the various kinds of things that we were up to, and get a greater understanding of how it might impact the overall Geo market. We had a fantastic conversation, and it was clear that Dan was an individual with an incredible sense of what’s up in the world of location.

So we had brainstormed just a bit and decided that, while it didn’t quite make sense to hire him at that time, we still wanted to have his mindshare around the office. What did we come up with?

Sponsor a desk

If you’re a company with a little extra space, and you know of some folks that are thinking about the same, or similar kinds of problems as you, but are independent contractors, or working on their own gig, do them a favor: sponsor a desk. It’s really easy to do…just write up some simple NDA’s, maybe some sort of contractor agreement that says that you’ll be trading some desk space for some time and you’re done.

In our case, in exchange for the desk space and access to our office, Dan agreed to give us one hour a week of consulting work. For us, it’s more than helpful. Our team is primarily engineers, and they all bring unique and interesting things to the table, but some of them are somewhat unfamiliar with the intricacies of the geographic world. So we are beginning to solve that with this setup.

Have brown bag lunches

Today we began an experiment, one that would bring in experts in our field periodically to help think through some of the unique issues that we may be facing, as well as just get a general sense of what that expert might be facing. So we brought in all of our current and future employees, and even had our SF engineer extraordinaire, Mike Malone join us via video.

Dan has spent the time informing the team about the world of Geographic Information Systems, and helping us understand some of the basic concepts that the GIS world has known for years. We solve some of the various problems he talked about in very different way than most of the GIS community, so it’s very cool to see how people have done it for years, and what our work and contribution might mean for the future.

We’re certainly excited to see what comes of this “partnership”, however informal that it may be. We’ll also be very inclined to recommend this setup to others, as well as try bringing other individuals over time as it makes sense. It’s been a lot of fun so far.

Alright folks, it’s that time again and we’re hiring!

We’re looking to hire some rock stars to join up with the SimpleGeo team in Boulder, CO or San Francisco, CA and help build our business even bigger. If you’re a Biz Dev hustler or Developer Advocate / Relations guru, we’d love to talk to you!

Here’s the gist:

Business Development

• At least 3-5 years making business development or similar deals
• Self-starter
• Ready for the “startup world”
• Able to build and maintain meaningful business relationships

Developer Advocate / Relations

• Must love developers
• Needs a solid engineering basis (so that you can…well…relate to developers)
• Able to respond to support requests and identify problems
• Engage businesses on our platform to understand wants, needs, and what we can improve.

If any of these descriptions sounds like you, please get ahold of us! Email us at jobs [at] simplegeo [dot] com!