Big Data visualizations using crossfilter and dc.js

Big Data visualizations using crossfilter and dc.js

Crossfilter and dc.js are awesome libraries. They are pretty neatly coupled and let you create interactive visualizations for the web. The problem arises when you're dealing with massive data. Memory limits of the browser is hit, the filtering becomes too slow and it becomes impossible to handle such datasets. What we can do, is move the loading of the data and the computationally intensive filtering, on the server. So you have your charts rendered using dc.js on the browser, everytime a user interacts with the charts we send an AJAX request to the server to compute the filter, the results are obtained as a response to that AJAX call and dc uses them to render the results. Sounds great! But how do we go about implementing it? Well lets see.

The Server

This is where we define our crossfilter dimensions, usual stuff, no big deal. Only difference is you use it in node instead of your browser.

Cool so we've done the basic stuff, now how do we handle the filtering requests, which are gonna happen via AJAX requests.

The communication

Now this cool idea has been borrowed from a friend of mine, Ziggy Johnson, who made an AJAXified version of the crossfilter demo. What you do is encode the filters in the form of a GET request. Now on filtering, you make an AJAX call to the server using the string containing information about the filters, the server applies those filters and returns the filtered result sets back to the client.

The client side

So we have the server and the communication mechanism set up. Now we need to code the client. The challenge is dc.js's coupling with crossfilter. Now dc.js charts are built on top of crossfilter dimensions and groups. We have moved all crossfilter related things to the server. So we create, what are called "fake dimensions" and "fake groups". These are nothing but objects that have a filtering function which in turn makes the AJAX request to do the actual filtering on the server.

Similarly create fake groups.

 

References

  1. Ziggy Johnson's Crossfilter AJAX demo http://flights-ajax.herokuapp.com/
  2. dc.js FAQ for fake dimensions and groups https://github.com/dc-js/dc.js/wiki/FAQ
  3. My answer to a stack overflow question http://stackoverflow.com/a/24541415/502835
  4. My question on stack overflow http://stackoverflow.com/questions/24184986/using-dc-js-on-the-clientside-with-crossfilter-on-the-server/

About Ganesh

Page with Comments

Leave a Reply

Your email address will not be published. Required fields are marked *