Thoughts on IFTTT
Riverbank In Moonlight, Charles-Francois Daubigny, 1875
I listened to an excellent podcast on IFTTT architecture a while back. There are a lot of interesting tidbits in the podcast, but I found these three things most fascinating about IFTTT.
1) Their software is largely based on polling rather than pushing
2) They are heavy users of Apache Mesos for scheduling
3) Most (all?) of their code is written in Ruby
Polling
Until recently, I had not spent a lot of time integrating into third-party services and assumed that each of these serviced had some event-based APIs (webhooks) you could subscribe to. For example, my Facebook feed. I could subscribe to a raw event stream of my updates and as updates occurred, they would publish an event and I could consume that event. I learned the hard lesson that this is often not the case. Instead, I can get an API of events that happen on Facebook, but I need to check to see if they have updated over some time window.
A representation of Poll vs. Push Architecture
This is much harder to coordinate and inherently much more expensive than the alternative. However, because of the broken and non-collaborative approach of the internet, there is not much else they can do. Effectively, IFTTT is constantly polling every user's data from APIs to check for the events they have asked to watch for. This is incredible when you think about someone like me. For a while, I had about 25 IFTTT recipes that watched for new iPhones to be posted on eBay and published those items in a spreadsheet. The sheer number of polls for that seems alarming, but IFTTT is doing this for all of its users (Millions).
Apache Mesos
A representation of Mesos with a Zookeeper Quroum
Since IFTTT is constantly deploying processes that poll all the time, they have to think intelligently about how to do that, and Apache Mesos is a big part of how they are able to accomplish this. Kubernetes has “won” the orchestration platform wars, however, Mesos is still great at running many workflows that Kubernetes just isn’t good at yet (mostly job oriented). I’ve seen use cases like this for Mesos and it’s largely why running Spark, Flink and other distributed data engines are easier to manage on Mesos than on Kubernetes.
Ruby
IFTTT heavily uses Ruby. Although they are actively looking to migrate away from Ruby, they are still using it and have been able to scale to where they are now with it. I think in a world of “future proofing” software it’s cool to see something that works at the scale, reliability, and performance of IFTTT running on a platform most consider “slow.” Some might argue that IFTTT is slow, and you will see some latency whenever running some recipes. That being said, when I consider the whole of what is happening per recipe, I think that most latency is tolerable. It’s not really IFTTTs fault that the internet is so broken and they have to poll such commonly used services like Twitter and Instagram. Maybe a future where allowing users to control the flow of data into and out of applications in a secure way would help us generate much cooler stuff than we have now.