Over-engineering My Smart Home
Anyone into connected home devices has already felt the pain of having many great devices in isolation but no real fabric that glues them together. Over the last few months, I've worked, on and off on a project that attempts to be a customized fabric for connected home devices. This post outlines some of the trouble I've had and cool things I've discovered.
Smart Home
Before jumping into protocols and boring parts of the connected home, I wanted to talk about my motivation. My smart home is a combination of devices that all communicate via different protocols and are perhaps, smart all on their own, but they don't work as a system. After spending all the money and time I have on these devices, I've been obsessed with the idea of getting them to actually work together. At first, I thought I could stitch together logic with something like IFTTT, but that was a monumental failure. Then, I thought I could host my own server and automate from there with something like Homebridge. That was also more work than I wanted.
I ended up looking deeper into the protocols (Zigbee, Z-wave, HTTP, CoAP etc) behind all the devices and how I could take all the noise the devices make and stitch together a cohesive DSL for my smart home. In that search and experimentation, CoAP (a protocol) and Streamsets (a streaming platform) have been the most interesting.
Architecture
The overall architecture is insane for someone who just has 20 devices powering a "smart home." In order to get the home to actually be "smart," more enrichment and processing of event data needs to happen than what is available on most platforms. I'll walk through each piece of the architecture throughout the post.
Streamsets
I’ve been very impressed with StreamSets as an open source project. It fills an important gap in the streaming landscape as a plug-and-play platform. It's a very nice platform if you have a streaming pipeline idea and want to get it up quickly. I have a lot of ideas about how I want to utilize streaming in my smart home s, but I don't want to spend too much time in the weeds of a big streaming framework.
Rough sketch of how Streamsets works
Architecturally, Streamsets is pretty simple. It's essentially Spark Streaming under the covers with nice integrations built on top of it. You can deploy Streamsets as a stand-alone service, on top of a Spark cluster (Yarn, Mesos) or on top of Kubernetes. I typically opt for deploying it on Kubernetes, which is a non-trivial ordeal, but worth the extra control.
Running StreamSets on Mesos
CoAP
Constrained Application Protocol (CoAP) is not too interesting on the surface. It's embarrassingly similar to HTTP, with a much smaller footprint. It supports restful verbs (GET, POST, PUT, PATCH, DELETE), headers, and some other functionality.
CoAP Packet size and methods versus HTTP packet size and methods
With that small size, you can do things like communicating directly from one small IoT device to another without stressing about the performance implications. CoAP also works over the internet and without it (Devices can communicate to each other like SMS).
A modern approach to inter-device communication. All IoT devices send messages to some centralized server to communicate with each other.
In my smart home, very few devices support CoAP (3 of 20) and it's not necessarily popular outside of devices like the Raspberry Pi. That being said, I've had success doing small things with CoAP, but the things that have worked have been super encouraging. I've got my Raspberry Pi to speak directly to a CoAP open-close sensor, and my Raspberry Pi to speak to an Arduino based connected device as well.
What I've found from using CoAP is that it's much better as a protocol to send packets from connected devices to a Hub like SmartThings than it is to speak directly to other devices. The coupling has very few advantages (decreased latency) and a host of disadvantages.
CoAP is a sufficiently small protocol that communication directly to and from IoT devices is possible. This rarely occurs in the real world, however.
Streaming
After data goes from HTTP, ZigBee, Z-wave or CoAP to my hub and then to my Streamsets cluster a lot happens after that. I have written some custom code to strip out useless bits of information from the message packets and enrich the data coming through. This way, I can make some more intelligent decisions about building a DSL. Effectively, though messages are coming in via non-streaming platforms, they are streamed out (actually micro-batched) of Streamsets and into several sinks. They are either sent to a file server (videos, audio clips etc), and object store (for building models) or back onto the event hub to trigger events. I am able to transfer large files, implement custom logic and trigger events all from one place without writing tons of code.
Wrap Up
This project is a bit of a headache, but I am converging on a good prototype. Once completed, I plan on sharing a full set of requirements to get this all working as well as the DSL I've come up with for handling it.