Introducing scalding-dataflow

For the last 3 days, I've been working on trying to understand the Google Cloud Flow pipeline semantics for batch processing. Result was a ScaldingPipelineRunner for DataFlow pipelines.

NOTICE (You've been warned)

1. It is still in very very early stages. 
2. It doesn't have all translators implemented (as of writing)
3. It isn't tested on a Hadoop Setup yet

It runs WordCount though :) Do give it a spin. 

Github - https://github.com/ashwanthkumar/scalding-dataflow

It goes well with scalaflow - Scala DSL for building dataflow pipelines. 

Special thanks to Cloudera's spark-dataflow project. Couldn't have done without it :)

User ID Normalization from Big Data book using Scalding

Thanks to my mentor, I have been following Big Data book closely in MEAP. On Chapter 6, he talks about the User Identifier Normalization problem, which is an iterative graph algorithm. He provides a reference implementation using Cascalog.

Since I have been using Scalding close to a year now. I re-wrote the same in scalding. This is my first attempt in writing an iterative algorithm using scalding.

Any kind of feedback is highly appreciated.