Introducing CrowdSync

Crowd-what?

This year, Nick and I wrote a little CoffeeScript program along with some HTML, CSS and a bit of C# for good measure. If you visit a web page that this code is running on, it makes your phone blink a randomly assigned color at specific times throughout a song performance. Your phone becomes synchronized with many other phones blinking the same or different colors all in time with a musical track which is also being played by a live band. We call it CrowdSync.

The Challenge

In August this past summer, Kim Vehon (from Central’s worship team) came into Nick’s and my office. She showed us this video of cell phones blinking a single color and emitting a tone to create music. It was a pretty cool idea, but we weren’t too sure where she really wanted to take it. I think initially, she wanted it as some part of the worship service element. Having phones somehow incorporated into the set that blinked with the music.

Initially, this sounded kind of boring to Nick and I. We thought, “What if we could make everybody’s phone blink like this, rather than just a few phones on stage.” An interactive worship experience sounded much cooler than a fancy set piece.So that’s where this whole thing began. And it just snowballed from there, becoming bigger and cooler as we got closer and closer to Christmas.

Mr. Negativity

In the project’s earliest stages I had convinced myself that there is no way that we’re going to pull this thing off. Early on, I was Mr. Negative Nancy about the whole thing. The core problem we were trying to solve was what we geeks call non-trivial. That is to say, we had a mountain of a problem to climb: We needed to make everybody’s phone blink at the same time. Not just roughly the same time. We needed them to blink at precisely the same time, with millisecond precision.

If you’ve never thought about this, here are just a few of the variables we found ourselves facing:

• Inconsistencies in how time is kept on each device.
• Time differences between carriers and even between devices on the same carrier.
• Different operating systems and mobile browsers and the limitations they imposed.
• Network latency between the phone and the server.
• The complete lack of control afforded to mobile browsers by the client OS.
• A great myriad of things that can happen whilst dealing with a phone (i.e. - What happens if somebody gets a phone call, or a text, or their phone goes to sleep?).
• What’s going to happen when a thousand people hit our web server all at once?

So you see, being the pragmatic geek that I am, I naturally thought there was no way we could pull this off. Boy was I wrong!

The Solution… Round 1

A couple days later, Nick went into a kind of fit of inspiration that would ultimately prove me wrong. I’ve been thankful for that every day since. :)

We initially talked about using something like Node.js with Socket.io to send push notifications to clients teling them when to “play”. We decided fairly quickly that this approach would be too “chatty”. We needed something that would be fairly quiet on the network to help solve some of our network scalability challenges.

Nick came up with the idea to start from a known time source, the server’s time. We wrote a little C# web service with 2 methods. One method returns the current time, and the other returns a designated start time based on a campus ID passed to it. We would then interact with these services on the client via AJAX.

This solved the time inconsistency problem. But how do we deal with network latency? We ended up prototyping out a little bit of CoffeeScript that would poll the server several times asking for the current time. We then took the server’s time and compared it to the client’s time. We did this over and over trying to get the smallest variance between the client and server possible. Approximately 20 tries seemed to be the magic number of times to poll the server. This allowed us to kind of synchronize our watches with the server and get each of our clients doing things at the same time. Since the client time becomes relative to the server’s time, in theory, this technique could even work across time zones.

OK, so we had the time problem down. Now we needed to know when to “play” our “note”. So we set up a timer on the client that would check the current time against the known start time for our “song”. We needed this to be very responsive, so we settled on doing this check 100 times a second. So like a little kid with a full bladder on a road trip, our little app asks “Are we there yet?” incessantly until it’s time to play the note it’s been assigned.

It took us about a full day to get a working prototype. And at the end of that day, we had something that would take an array of JavaScript object literals and play each note. We had all of our IT team’s phones in our office all blinking in perfect unison. It was brilliant.

More Problems… More Solutions…

Now that we had a working prototype that solved our core set of challenges, we needed to address another core piece of the puzzle. How will we get the musical arrangement into a format our app could understand?

After some pestering, we were finally given a midi file exported from an mp3. It was the piano arrangement that would be played during the service, played one note at a time. Thankfully, Nick found a midi parser written in C# that was able to decode the midi file and convert it into an array of relative timestamps. So we could convert this timing metadata stored in the midi track into an array of JavaScript object literals. The server determined the start time of the performance, and each note had a time stamp associated with it that was relative to the start of the track.

We then took this data and converted it to our “adjusted” time on the phones. We could add this relative time to the start time of the song and then our phone clients would know when to play their note during the course of the performance.

Our next challenge was getting the light boards and band to fall in step with our blinking lights. Our lighting genius, David Empy, suggested we use QLab to fire the lights and the click track for the band. We ended up using QLab for the lights only and created a control panel that synced up the QLab Mac’s time in much the same way that the phones became synced. This control panel allowed us to set start times for the performance at one of our campuses.

To synchronize the worship band with our app, we decided to fire the click track ourselves using WebKit’s Audio API *. After working through a few performance issues, we settled on a method of pre-loading and running the audio in a way that would fire at exactly the right time. We then were able to use an offset to account for any lead-in time on the click track for the band. Not super straight forward, but it worked really well.

Creating the song visualization

This was probably the funnest part of the whole process for me. About a month and a half ago, Nick and I found out we were going to be responsible for creating a visualization to go on the side screens. This first started out as a grid of rectangles. Each rectangle represented a phone in the grid. This allowed us to create some pretty neat looking patterns, but it was ultimately limiting. Our worship team didn’t think it was “Christmasy” enough (they were right).

So we went back to the drawing board and started playing around a little bit with the HTML 5 canvas element in conjunction with various pieces of audio API. We came up with programmatically drawing the Luminous graphic with code on the canvas and animating it with our CoffeeScript.

I enjoyed this part the most. Nick ended up coming up with the math that would plot each point of light on the tree (using some Geometry calculations that I don’t think either of us have used since high school). We assigned the color randomly and filled it with a radial gradient.

We even worked on adding little subtle touches like fading the lights out rather than shutting them off abruptly, moving the center point of the radial gradient for each light around and adding a bit of flickering here and there to make the tree look more organic.

Gotchas

Initially, syncing with an NTP server (like time.apple.com or time.windows.com) sounded like a no-brainer. We learned fairly deep into the project, however, that is can throw a wrench in things pretty quickly. If all your clients are dependent on time, what happens if the server’s time drifts?

Q: What happens when the server receives an NTP update and adjusts its time accordingly?

A: It starts serving up a new time to clients, that’s different from original time. This is not good.

We also ran into this same problem on our QLab Mac that would fire the lights, and once again on our iMacs that ran the visualization on the side screens in the house and fired the click track for the musicians. Yep. Still not good.

It initially looked like “gremlins in the system”, but we noticed that once we disabled NTP syncing and just had the clients keep time locally, our results became much more consistent. Since each client could adjust it’s start time based on the delta calculated off the server’s time, we could still maintain our synchronization between devices without relying on NTP.

Future iterations

Currently, all of our performance start times are accessed via an ASP.NET web service and stored as Lookups in our Arena ChMS database. We’re going to be breaking this dependency in the next major release. We prototyped it out using Arena since it offered us many conveniences, but we’d like to make it more portable and not reliant on a proprietary ChMS product.

We’ve got plans to re-write the back end of the system using Node.js and Express to offer a RESTful API to access the relevant information based on campus. We’ll likely be storing the data in a No-SQL database (MongoDB is looking pretty good) or perhaps something like MySQL if we end up needing a more traditional relational database engine.

I’ve got some other ideas that are perhaps a bit too abstract to go into here, but we’re actually looking to make CrowdSync more of a real-time app. We’ve got some pretty exciting ideas that we’re looking forward to exploring that should make the system better and more stable in future releases.

Conclusion (TL,DR)

Even though I was convinced we couldn’t pull it off in the earliest days, we did it! We successfully synchronized many devices to blink colors at the exact same time. Not only did they blink their assigned color at exactly the right time in time with each other, we were able to make them do it in time with music being played by a live band.

It was a huge success and we are super blessed to be a part of such an awesome team that could pull something like this off.

Keep an eye on the GitHub repo we’ve set up. We’ll be releasing the source code in the next few days. We’re also working on a video to more easily demonstrate what the app is capable of doing. As soon as we’ve finished work on it, Nick and I will make sure it gets included in the README on GitHub (and on our blogs too).

Pop over to Nick’s blog and give his post a read too. 

* I actually liked Mozilla’s HTML 5 Audio API better, but Google’s V8 JavaScript engine was just too fast for us to drop Chrome completely for this project. When you’re ‘ticking’ every 10 milliseconds and painting a <canvas> or manipulating DOM elements, you really need to eek out all the performance you can get.

Building a Web Client like Pros

Updates at the bottom

In this post, I will guide you with some tips and tricks on how to build a web application with the top of your own API.

So, let’s say you have an Awesome Web Startup and you want to build it the modern way. What I mean is: You have an API and a User Interface “Javascript application” that communicates with the API . We can find dozens of websites that uses this technique, for example Twitter, Foursquare, Quora and much more.

Lets say you are having myawesomeapp.com as a front end for your users. Once they signed in, they grant authentication to access the API. Then you want to make Ajax requests to other server than the domain that hosts the client application e.g. api.myawesomeapp.com. This leads lots of the newbie startups to setup the API server in the same domain of the client application to avoid the limitation of XmlHttpRequest object in the web browsers: myawesomeapp.com/api. This problem is caused because Same Origin Security Policy of browsers, which prevents a page from accessing data from another server. I will not talk about the Same Origin Security Policy issue, you can read about it here.

Leading startups like Twitter, are using their own public API in their client applications which is in most cases hosted on a different server than the application. I will take Twitter as a case for how they make call requests to their API server api.twitter.com and the way of the authentication they adopt for communication between twitter.com and api.twitter.com. Thought, Twitter is using oAuth to authenticate users for their API externally.

The easy way of that is to grant them the authentication by sharing same cookies of the front end application. By being signed in and authenticated to the client application, the API server will be accessible to the user via the client.

For each user that signed up for Twitter, the system generate a unique token for each user (in case if Twitter is an oAuth client: access_token for the internal oAuth application). This token is being saved in the cookies with key auth_token. The token is used to grant the access for the user for to the API.

Talking about granting users’ access to the API could be done by many ways, cookies as Twitter does, sending the token as a parameter in the requested url, or sending the token via a header. Since this token is a randomly generated and created, beside it is unique token cross all users, it leads us to be hard to guess string. So on the API side, we find the user with this unique token and give her the access to the API.

I use Rails for building my Javascript Client and Grape for the API. In the rails application I use plataformatec/devise for authentication, which is built on the top of hassox/warden. So in the API I can access the user via env["warden"].user.

We also share the cookies of the rails app with the API by:

# in config/initializers/session_store.rb
App::Application.config.session_store :cookie_store, key: '_app_session', :domain => :all

In the API’s config.ru

# THE_LONG_SECRET is from config/initializers/secret_token.rb in the rails app
use Rack::Session::Cookie, key: "_app_session", secret: "THE_LONG_SECRET", domain: ".domain.ltd", httponly: true

So now the cookies are shared by the two apps: the api and the client.

Once the user signed in, the authentication can be checked through env["warden"]. But we want to focus now on how to authenticate them throw the authentication_token.

In Devise, we have a great feature that we can generate a unique token for each user, just add :token_authenticatable to devise modules list in your model (User):

devise :database_authenticatable, :registerable,
         :recoverable, :rememberable, :trackable, :validatable, :token_authenticatable

Make sure that Devise is also ensure that each user has an authentication_token with adding before_save :ensure_authentication_token to the model.

Also, you can add the user’s authentication_token to cookies with key auth_token by a Warden callback (And another callback to remove it from cookies after signing out):

Warden::Manager.after_set_user do |user, auth, opts|
  auth.cookies["auth_token"] = {
   value: user.authentication_token,
   domain: :all,
   httponly: true
  }
end

Warden::Manager.before_logout do |user, auth, opts|
  auth.cookies.delete("auth_token", {domain: :all})
end

In the next snippet of code, I show you how we authenticate users to the API. So, as you see we grant the authentication via warden and authentication_token via cookies, headers, and as a parameter.

module ApiAuth
  # Finding auth_token in the HTTP_COOKIE.
  # Will loop throw all cookies to find it.
  # Returns the value of auth_token when find it.
  # Else returns nil.
  def auth_token_with_xml_http_req_via_cookies
    if env["HTTP_COOKIE"]
      env["HTTP_COOKIE"].split(/; /).each do |cookie|
        k, v = cookie.split(/\=/, 2)
        if k == "auth_token" and env["HTTP_X_REQUESTED_WITH"] == "XMLHttpRequest" # We check also if the request was made by an Ajax request
          return v.to_s
        end
      end
    end
    return
  end

  def auth_token_via_headers
    if env["HTTP_X_AUTH_TOKEN"]
      return env["HTTP_X_AUTH_TOKEN"].to_s
    end
    return
  end

  def warden
    env["warden"]
  end

  def authenticated
    if warden.authenticated?
      return true
    elsif auth_token_with_xml_http_req_via_cookies and User.where(:authentication_token => auth_token_with_xml_http_req_via_cookies).first
      return true
    elsif auth_token_via_headers and User.where(:authentication_token => auth_token_via_headers).first
      return true
    elsif params[:auth_token] and User.where(:authentication_token => params[:auth_token]).first
      return true
    else
      error!({ code: 401, message: "Unauthorized." }, 401)
    end
  end

  def current_user
    warden.user ||
      User.where(:authentication_token => auth_token_with_xml_http_req_via_cookies).first ||
        User.where(:authentication_token => auth_token_via_headers).first ||
          User.where(:authentication_token => params[:auth_token]).first
  end

  def authenticated_user
   authenticated
   error!({ code: 401, message: "Unauthorized." }, 401) unless current_user
  end
end

So, in the api you can ensure if the user is authenticated by authenticated_user method, and you can access the user object by current_user.

Now, the beautiful part, you are on the client side myawesomeapp.com and want to make Ajax requests to the API api.myawesomeapp.com that returns JSON object.

I’m using jQuery as a javascript library, and here is a simple GET request:

$.get("http://api.myawesomeapp.com/1/me")

Ops!

XMLHttpRequest cannot load http://api.myawesomeapp.com/1/me. Origin http://myawesomeapp.com is not allowed by Access-Control-Allow-Origin.

There are lots of technique to avoid this limitation of browsers (mentioned at the top of the post) you can read here.

The best working solution is to setup an iframe that served by the API domain and make requests to the API via it. I read lots of Javascript in the source code of Twitter and Foursquare and I’ve extracted a great piece of code to solve this.

First, set up a static html page receiver.html that served by the API server: api.myawesomeapp.com/receiver.html and contains this html code:

<html>
  <head><meta http-equiv="Cache-Control" content="public, max-age=31556926" /></head>
  <body><script>document.domain='myawesomeapp.com'</script></body>
</html>

Second, create an iframeManager in the client app (extracted from foursquare.com). Here’s a the coffeescript snippet of the iframe manager: (javascript version: https://gist.github.com/1555068)

iframeManager =
  xhrCallback: null
  iframeLoading: !1
  loadQueue: []

  addLoadCallback: (a)->
    if iframeManager.isLoaded() then a() else iframeManager.loadQueue.push(a)

  runLoadCallbacks: ->
    a() for a in iframeManager.loadQueue

  isLoaded: ->
    null != iframeManager.xhrCallback

   buildIframe: (apiServer) ->
    if not iframeManager.iframeLoading
      iframeManager.iframeLoading = !0
      a = document.createElement("div")
      b = "#{apiServer}receiver.html?parent=#{encodeURIComponent(window.location.href)}"
      a.innerHTML = """<iframe onload="window._tempIframeCallback()"  tabindex="-1" role="presentation"  src="#{b}"></iframe>"""
      c = a.firstChild
      window._tempIframeCallback = ->
        delete api._tempIframeCallback
        iframeManager.xhrCallback =
        if window.XMLHttpRequest and ("file:" isnt window.location.protocol or not window.ActiveXObject)
        then ->
          new c.contentWindow.XMLHttpRequest
        else ->
          try
            return new c.contentWindow.ActiveXObject("Microsoft.XMLHTTP")
        iframeManager.runLoadCallbacks()
      document.body.appendChild(c)

window.iframeManager = iframeManager

And here’s a snippet of the Api class that through it we will create requests to the API (javascript: https://gist.github.com/1555071)

At the bottom of the code you have to set the API domain and the API version: window.api = new Api "http://api.myawesomeapp.com", 1

class Api
  constructor: (apiServer, apiVersion) ->
    @iframeManager = iframeManager
    @apiServer = apiServer
    @apiVersion = apiVersion

  ajax: (req) ->
    if @iframeManager.isLoaded()
      req.xhr = @iframeManager.xhrCallback
      req.crossDomain = not 1
      req.dataType = "json"
      $.ajax req
    else
      self = @
      @iframeManager.addLoadCallback ->
        self.ajax req
      @iframeManager.buildIframe(@apiServer)

  request: (options) ->
    options.type = options.type or "GET"
    options.url = if options.url then "#{@apiServer}#{@apiVersion}#{options.url}"
    options.data = options.data or {}
    options.headers = options.headers or {}
    @ajax options

window.api = new Api "API_DOMAIN", API_VERSION # notice to change API_DOMAIN and API_VERSION

So now you have access to the api variable in the whole of the client application. To make requests to the API you have to use api.request() function.

Here’s some example how to make requests:

api.request({url: "/me"}) // this makes a GET request to http://api.myawesomeapp.com/1/me

You can use jQuery methods of $.ajax: done, fail, always

api.request({url: "/me"}).done(function() { alert("success"); }).fail(function() { alert("error"); }).always(function() { alert("complete"); });

We grant the authentication of these requests by the authentication_token that we save in the cookies.

if you have not authenticated via cookies you can use the header:

api.request({url: "/me", headers: {"x-auth-token": "xhg7t37vy3rvFS4jt4"}})

or by parameter:

api.request({url: "/me", data: {"auth_token": "xhg7t37vy3rvFS4jt4"}})

for POST requests:

api.request({url: "/articles", type: "POST", data: {...}})

for PUT and DELETE requests: we handle it the Rails way: make a POST request with _method parameter for the request type and don’t forget to add this line to config.ru in the API application use Rack::MethodOverride

api.request({url: "/articles/123", type: "POST", data: {_method: "put", …}})

api.request({url: "/articles/123", type: "POST", data: {_method: "delete"}})

I read this tweet yesterday, retweeted by a friend: Samer Abu Khait.

not having an API in 2012 is like not having a website in 1998.

— Brent Grinna (@brentgrinna)

January 2, 2012

You know how to handle authentication and cross domain requests. It’s your time to craft your application the professional way!

1- Dont forget to put this snippet you the top of your Client application

    <script type="text/javascript" charset="utf-8">
      document.domain = 'myawesomeapp.com';
      <!-- OR document.domain = '<%= request.host_with_port %>'; -->
    </script>

2- (Recommended step) You can drop using Warden in your application, and use only authentication via the cookie token, Also, I recommend to use CommonCookies rack middleware in case you dont want to set the domain manually (better for testing locally)

# THE_LONG_SECRET is from config/initializers/secret_token.rb in the rails app
use Rack::Session::Cookie, key: "_app_session", secret: "THE_LONG_SECRET", httponly: true
use Rack::CommonCookies
# dont for get to install `rack-contrib` gem and require "rack/contrib/common_cookies" in the top of API config.ru 

Web Links

Running a little less often during the holiday season but here’s what I’ve found interesting lately:

Here’s a link to a free coffescript ebook:

http://autotelicum.github.com/Smooth-CoffeeScript/SmoothCoffeeScript.html

One on jQuery fundamentals:

http://jqfundamentals.com/

Intro to the Dojo javascript toolkit (part 1 of 2):

http://net.tutsplus.com/tutorials/javascript-ajax/dig-into-dojo-dom-basics/

Getting started with MongoDB:

http://net.tutsplus.com/tutorials/databases/getting-started-with-mongodb/

Backbone.js tutorial (part 1 of 3):

http://coenraets.org/blog/2011/12/backbone-js-wine-cellar-tutorial-part-1-getting-started/

Places to look for jQuery plugins:

http://dailyjs.com/2011/12/06/jquery-roundup/

A mobile date picker:

http://baudson.cute-ice.de/serendipity/index.php?/archives/94-Introducing-Mobi-Pick.html

Creating a color palete (checkout the links at the bottom of the article for more posts on web design use of color):

http://designfestival.com/creating-a-color-palette/