hook.io

Nubstep: Overview and Part I - knob.io
Nubstep: Overview and Part I - knob.io

Nubstep is the project @pedrogte, @telmofcosta and me presented at Codebits 2011.

It is a sound synthethizer that collects inputs from an arduino, and generates sound through Core Audio. It also includes a web console for looking at readings (not shown at the competition presentation due to presenter lameness). All the data flowing from the arduino to the synthetizer is channeled through hook.io.

Overview:

image

  1. Arduino: Collect data from analog inputs (Detailed in this post)
  2. knob.io: Get readings from arduino and send them to hook.io (Also detailed in this post)
  3. knob-view.io: Show arduino readings on a browser
  4. nubstep: Synthetize sound waves and feed them to roar
  5. roar: Node library to output sound through Core Audio

Arduino Sketch

Reading the analog inputs on the arduino and sending them through the serial port is fairly easy. Here’s the arduino sketch:

(If you’re an arduino newbie, you can read “Serial port” as “USB”)

void setup() {
  Serial.begin(9600); 
}
 
void loop() {
  int analogValues[6];
  int i;

  for(i=0; i<6; i++)
    analogValues[i] = analogRead(i);

  for(i=0; i<6; i++)
    Serial.println(analogValues[i]);
  Serial.println();

  delay(100);
}

This code may seem familiar, and it is, because it’s basically this arduino example adapted to read all the 6 analog inputs.

Here’s the fritzing sketch for our box:

image

And the actual box:

image

knob.io

Reading the serial port on node.js requires the node-serialport module. Currently this module is stable enough, so, the hard work consists on interpreting the arduino output correctly.

Here’s code for reading the serial port and sending it to hook.io:

(See below for more details)

#!/usr/bin/env node

var util       = require("util");
var serialport = require("serialport");
var Hook       = require('hook.io').Hook;

var SerialPort = serialport.SerialPort;

var SerialHook = exports.SerialHook = function (options) {
  var self = this;

  Hook.call(this, options);

  this.on('hook::ready', function () {
    var sp = new SerialPort("/dev/tty.usbserial-A7006xos", { 
      parser: serialport.parsers.readline("\r\n\r\n") 
    });

    var first_time = true;
    var values = [-1, -1, -1, -1, -1, -1];
    sp.on("data", function (data) {
      // always ignore the first sample
      // data may be incomplete -> no way to know
      if(first_time) { first_time = false; return; }
      var knobs = data.split("\r\n").map(function(k) { return parseInt(k,10); });
      if(knobs.length !== 6) { return; }
      var changes = 0;
      knobs = knobs.reduce(function(map, k, i) {
        if(k !== values[i]) { map[i] = values[i] = k; changes++; }
        return map;
      }, {});
      if(changes !== 0) {
        self.emit('knobs', knobs);
        console.log('Emitting: ' + util.inspect(knobs));
      }
    });
  });
};
util.inherits(SerialHook, Hook);

var serialhook = new SerialHook({
  name: 'serial',
  'hook-host' : '0.0.0.0'
});
serialhook.start();
Reading the first valid sample

The data stream sent by the arduino has the following structure:

...
<analog input 0>(end of line -> \r\n)
<analog input 1>(end of line)
<analog input 2>(end of line)
<analog input 3>(end of line)
<analog input 4>(end of line)
<analog input 5>(end of line)
(empty line, which means just an end of line)
<analog input 0>(end of line)
<analog input 1>(end of line)
<analog input 2>(end of line)
<analog input 3>(end of line)
<analog input 4>(end of line)
<analog input 5>(end of line)
...

Actual data looks like:

...
511
255
767
1023
0
511

511
239
767
1023
0
511
...

To read input from the arduino, node-serialport is initialized with:

    var sp = new SerialPort("/dev/tty.usbserial-A7006xos", { 
      parser: serialport.parsers.readline("\r\n\r\n") 
    });

The parser option is a recent feature of node-serialport and used like this allows us to split readings from the arduino easily. In this case, note that the empty line between readings results in two consequent end-of-lines on the data stream (one from the last input being read and the other from the empty line).

Using the parser option simplifies parsing a lot, so, getting each reading becomes just wait for data with an event listener:

    sp.on("data", function (data) {
      //...
    }

There is no flow control nor any field delimiters (apart from the line endings). This means that the application reading the arduino feed may start reading it on the middle of something, so, the first input reaching the app (for the data given above) could be:

23
0
511

511
239
767
1023
0
511
...

The the first number is a truncated 1023. This is why the code always ignores the first set of readings, even if it contains the expected 6 readings.

Reading 6 and only 6 inputs

The data transmission protocol presented here is too simple, not only there are no field definitions, packet delimiters, etc, but there are also no checksums. This results in the following stream being received from time to time:

...
511
255
767
1023
0
511 // Note no empty line between these values.
511 //
239
767
1023
0
511
...

This kind of error results in a stream of 12 values being read instead of 2 sets of 6 values. Due to the nature of this application, ignoring this values is safe, so off they go.

Caveat emptor: If you’re running this code, check for the device used to initialize SerialPort - it may be different according to the arduino version that you’re using, so make it the same as the one that’s configured on the arduino IDE.

Sending data to hook.io

To define a hook in hook.io you start by extending the Hook class:

var SerialHook = exports.SerialHook = function (options) {
  //...
};
util.inherits(SerialHook, Hook);

To boot hook.io, instantiate the hook and start it:

var serialhook = new SerialHook({
  name: 'serial',
  'hook-host' : '0.0.0.0'
});
serialhook.start();

The actual hook code relies on calling the super constructor and waiting for a hook::ready event:

  // ...
  Hook.call(this, options);

  this.on('hook::ready', function () {
  // ...

For our data protocol we chose to only send value variations, so the following snippet deals with detecting those changes and sending them encoded as maps to hook.io:

      var changes = 0;
      knobs = knobs.reduce(function(map, k, i) {
        if(k !== values[i]) { map[i] = values[i] = k; changes++; }
        return map;
      }, {});
      if(changes !== 0) {
        self.emit('knobs', knobs);
      }
CQRS and Hook.io

Hook.io is popular library for node.js, which allows you to build up distributed cluster of applications (hooks) which communicate by sending an events to each other. Command and query responsibility segregation (cqrs) is the way to design your server, where write and read side are completely separated. In last few weeks I’ve found those two perfectly match each other, allowing you to build up decoupled, scalable and fault tolerant applications. I expect reader to be familiar with both hook.io and cqrs. If you’re not, links above would be a great start.

Look into architecture

To build up your application, you would split it into multiple pieces - hooks. Each hook is a process with a single responsibility (the pattern is used in OOP as single responsibility principle, although its well suitable on the processes level as well). Each hooks does its job and emit event if something happens. Another hook, can listen for an event and act on that. Its basically observer pattern and thats the way hooks talk to each other (inter process communication is implemented in hook.io and is based on dnode).

In case of CQRS system, the application was distributed into 3 main hooks:

Architecture diagram

Fig.1: Green boxes (webserver / domain / denormalizer) represent separate processes (hooks). Each of them works separately and send events (green arrows) to other components. As you can see, events and read view data are stored separately.

Domain

Domain is responsible to encapsulate the business logic. Its a write side of the application stack.

Domain architecture

Fig.2: Domain architecture. The domain has single command bus and multiple handlers (one per each command). Each handler can work with multiple aggregates and orchestrate processing of the command.

Domain listen for a specific set of hook events, where each trigger one command. Command is sent to a command bus, where appropriate handler is chosen. Handler executes the command by calling public API of aggregates (business unit in CQRS system) and each time command result in change of the system state, aggregate emits an event (This pattern is called event sourcing), which is stored in the event storage and emited from domain hook for interested parties.

Denormalizer

Data in the event storage would be hard to use to build up the views. Transforming the events to “read-ready” data (the data displayed in UI - reports, forms, …) is the job for denormalizer. Each time new event arrives, it takes the last snapshot, replay the new event and store the new snapshot (there can be even multiple snapshots with different version being built in parallel).

Denormalizer architecture

Fig.3: The denormalization process can be done either synchronously (directly on new event arrival) or asynchronously (sitting behind the queue, triggered once per X events or once in a given time) if you are ok with the data being stale for a reasonable amount of time. Both approaches can be used for different views.

Webserver

The webserver responsibility is to server the requests. It doesn’t matter which framework, protocol or data type is used. All what is done by webserver is processing the incoming request, build up an event with command to domain (in general when some data need to be saved) or query the read model (if some data need to be loaded from server). Notice the data were already denormalized for a read side, thus loading the data by webserver is trivial and very fast. It mostly does simple query by an ID to the read data storage.

Why cqrs and hook.io?

Scalability

In given configuration, you are able to scale each system component separately. You can choose a different strategy for each or use a different database for your events and for your read data models as well. With a hook.io you can easily distribute the components across your server infrastructure.

Fault tolerance

With a recent changes there is no master hook, thus the system keep running if any hook fail down. On top of that with an autoheal feature, system can try to heal itself by spawning new process after failure. Because hooks are very light-weight processes, they can be spawned in a very fast way, ensuring your system keeps up and running with a minimal down time.

Decoupling

You have well decoupled architecture where your delivery mechanism (webserver) and data persistancy (denormalizer and read model) are decoupled from your business logic (domain). The biggest benefit in my eyes is, that you are allowed to focus on a single thing in a given system component.

Easy webserver switch

Because webserver only responsibility is to handle requests, you would have no problem to change it for another one. It would make sense as well to have multiple webservers side by side. One can serve the HTML pages while other might focus on JSON API (if your HTTP server goes down, your API users wouldn’t worry at all).

Ability to change read data

It happens quite often these days business requirements change significantly in time. Data your users wanted to see might not be valid anymore thus you will have to adapt as well. Event sourcing allows you to drop the read data completely and reconfigure your denormalizer to build up another read data (as long as necessary information is contained in your events). This mean you can react instantly and project changes even to the past.

Conclusion

I hope I’ve listed reasonable set of arguments, why it might be worth to use cqrs with hook.io. I am not saying its silver bullet for everything but might be well suitable for many systems.

Project node cqrs (v0.5.1) is still quite a lot under development and would probably be considered as naive by C# or Java enterprise guys, where CQRS is used more often, although its light weight and supports the described architecture already. I am working on another set of features and if someone is interested in more details about implementation with hook.io let me know.

Hook.io: Hooking to Versatility

Hook.io[http://hook.io]

hook.io is a versatile distributed event emitter built with node.js

hooks are nodes which can seamlessly work together across any device or network to create self-healing meshes of i/o

Features :

  • Build applications intelligently by combining together small hook actors
  • Rapidly compose new functionality in a distributed and organized way
  • Dead simple Inter Process Communication built on-top of node’s native EventEmitter class
  • Supports a large amount of common i/o sources out of the box
  • Additional i/o support from a rich community and network of user-created hook libraries
  • Built-in auto-discovery systems using http, tcp, mdns, and hnet
  • Hooks can exist on any device that supports JavaScript (cross-browser support via socket.io)
  • Seamlessly manages the spawning and daemonizing additional hooks
  • Legacy applications can easily be extended to work with hook.io