You spent 3.5 years on this??
Muut Inc • 2013-04-26
When we started working on a modern forums and commenting platform in 2009, we certainly didn't envision launching it in April 2013.
When you first look at Muut, it doesn’t appear all that complicated. You login, you read text, post text, browse channels. On a busier day you might see a few more red dots and pop-up notifications… But that’s it. “How come this took so long to make?”
There are a lot of reasons. Mostly technical. Some psychological. (As in, we’re obsessed with this stuff.) And the laws of physics also came into play — there’s still only 24 hours in a day.
Muut is a far more complex of an effort than it might appear from an initial look. It seems deceptively simple on the surface, but inside there isn’t much that you have seen before.
Unlike any other forums/commenting, Muut is installed on a page by simply placing an anchor
<a> tag on the HTML source code. This is already something completely different — you won’t see anything similar out there. Typical solutions depend on IFRAME’s, but for us, this was out of the question. We wanted the forum to be an integral part of the site with a complete control over scripting and styling. This required quite a bit of thinking since it was an uncharted territory.
From the design and UI/UX point of view, just trying to forget what existing forum solutions were like took some time. The flaws were mostly obvious and many things could easily be solved individually, but coming up with a complete, simple and pleasing UI that still seemed somewhat familiar and welcoming to new users and then packaging it all with a modern backend into a working product was quite a task.
We wanted a user interface and experience that we considered absolutely ideal, with no compromises and no clutter. Designing and building a UX that comes even close to this is pretty difficult. We completely trashed the UX several times because of some minor flaws that wouldn’t be possible to fix without a complete restructuring.
Take channel sorting in the Muut admin view, for example: jQuery sortable alone weighs 42kb and depends on jquery.ui.core, jquery.ui.mouse and jquery.ui.widget. There was no way we’d double the size of our client because some small part needed sorting. So we rolled our own. We built our own tiny syntax highlighting library and our own communication libraries (instead of using socket.io). Same goes for tooltips, overlays, validators and custom select boxes.
Optimizing didn’t end with code, either. All the graphics — icons, smilies etc. — you see on Muut forums are made using css and an icon font (apart from the avatars, of course). Icon fonts are a great way of delivering infinitely scalable vector graphics in one tiny package, like a sprite. This is getting more common now, but in the early 2010 it certainly wasn’t a standard.
On top of all this — could we have imagined using ready-made icons? Not likely. So we created the font ourselves.
Which brings us to typography — it’s extremely important to us. Muut is based on text, after all. We couldn’t tell you how much time we spent looking for the perfect default typeface for the forums and our site. We don’t think we found them either, so there’ll be some developments in this area in the future.
We needed everything to be ultra-realtime. Totally beyond the standard Near-Real-Time. We needed a system that could handle all behaviors under load in 5ms or less (including all network overhead, and anything that happens asynchronously). That required some intense engineering.
Phone conversations typically have around 200ms latency. On Muut, from the moment an API call is first received until we send out the resulting notifications is about 2ms, once you add in the time for the message to reach you on the network (depending where you live), total time varies between 50ms and 80ms. This means that users will see the events you cause (posts, likes, etc.) faster than you can blink your eye.
We needed to be able to consolidate actions, initialization, and any behaviors into single API calls. We needed everything to feel snappy so this meant restructuring our API as we made improvements to the workflow. Since on the infrastructure side we simply can’t batch calls together because our performance requirements, it often required a great deal of engineering.
Current client frameworks and programming patterns are heavily dependent on REST, but that was not a good fit for our real time needs. We based our system on JSON-RPC which is quite different. We couldn’t find what pitfalls we’d encounter from reading about other people’s experiences – since there were none – so again we needed to write and think things from scratch. Oh — and we created our own version of the whole spec since the existing versions were too limited.
While the original goal was 5ms for median API response times, we ended up at 2ms on the final product.
Muut has to be capable of scaling to terabytes of data, as well as billions of posts a month. It’s essential that the requirements to scale to that level are built in from the beginning, and since we use Redis as our primary data store it required some rather complex engineering to make it work as well as some rather intensive benchmarking and testing with huge data sets to verify. You can read more about our Redis use here.
We also needed our infrastructure to be fully automated. We can’t have a huge ops team.
Here we are, now
If our goal was to simply build a commenting and forum thing quickly, we could have had it done in a few months. In fact our early proof of concepts/prototypes were ready within a few months. Most won’t have the requirements we had as they won’t be building large cloud based services like this.
They likely aren’t so obsessive over details as we are, either.
This post originally started as a forum thread.