Getting started with Druid: A high-performance, column-oriented, distributed data store.

Getting started with Druid: A high-performance, column-oriented, distributed data store.

Getting started with Druid: A high-performance, column-oriented, distributed data store.

The goal of this post is 2 folds: Being able to work with arbitrary data generated from http://www.json-generator.com/. Ingesting it into Druid. Performing filtering and aggregation on the data. We'll start with timeseries data and then try and look for ways to work with non-timeseries data. Generating data I used the following json spec to generate data from json-generator: Ingestion We generated a sample data from json-generator.com Now we need to generate the index file, which is like a metadata document that druid uses to ingest the files. The ingestion spec is a JSON document of the following structure: The

Continue Reading

Kue: Under the hood. Using the Job Que Externally.

Kue is a priority job queue backed by Redis, built for Node.js. http://automattic.github.io/kue I started using it in my projects where I was supposed to convert the results of computer vision algorithms into GeoJSON. These algorithms generate millions of segments and are run on very large images. Hence the conversion process takes a lot of time! My goal was to provide REST endpoints to users to upload their results and view them in GeoJSON. Kue fits perfectly in this setting since my application server was in Node.js. The users submitted their results which triggered the conversion job that kue tracked and

Continue Reading

Using DC.js with crossfilter on the server.

A while back I wrote Big data visualization with dc.js and crossfilter which proved to be pretty useful for a lot of people. In the post I sketched just an outline of the solution. For someone not familiar with the workings of dc.js and crossfilter it could be pretty vague. So I've decided to write a full working example of using crossfilter on a Node.js server to render dc.js charts. I've pushed the code on Github and over the next few days I hope to refine it. The README doc would contain details of things happening under the hood.

Continue Reading

Minimal React build environment

I've wasted a lot of time with Javascript build environments. I spend a lot of time trying to optimize my development environment in general. A lot of my projects use React for the fronted. This post is mainly for my reference to kick start a react project. Things I'm looking from my tool/toolchain: Transpiling ES6 and React to Javascript. Bundle all the component codes into a single bundle.js Have a "watch" mode to watch for edits on files SPEED! There's nothing worse than painfully slow builds! There is a whole array of posts and boilerplates available on Github. But most

Continue Reading

Javascript workflow/toolchain

I spend a lot of time writing javascript. Over the past few days I've tried to figure out the ideal workflow that works for me. I'm posting it here so that others struggling with similar issues can draw some inspiration and even help me improve my workflow. I use Node.js to do most of my javascript development, both frontend and backend. Lets break down the workflow into components: Terminal I recently switched to OSX and have started using iTerm which seems to be suiting all my needs. For Ubuntu I was using Terminator. I like being able to split my window

Continue Reading

Can computers imagine?

After a year of formally pursuing artificial intelligence and probably 2 more years of pursuit due to curiosity I went back to square one and am pondering upon more philosophical questions like:what is intelligence? I vaguely remember my first "Fundamentals of Machine Intelligence" class in which we discussed it as "mimicking humans". I had ofcourse underestimated the discussion then and was more keen on learning about advanced Neural networks and SVMs, a great folly indeed. But is "mimicking humans" all there is to intelligence? Surely there must be more. I looked around trying to find answer. A lot of introductory AI texts start

Continue Reading