[IDEA] Autoscaling in Hadoop

Everybody today uses Hadoop + some more of its ecosystem tools. Let it be Hive / HBase etc. I have been using Hadoop for writing production code from 2012 and for experiments much earlier than that. One thing that I don't see it anywhere is the ability to autoscale the Hadoop cluster elastically. Unlike scaling web servers having all map and reduce tasks full doesn't necessarily translate to CPU / IO metrics spiking on the machines. 

Figure - Usage graph observed from a production cluster
hand drawn on white-board

On this front - Only Qubole guys have seem to have done some decent work. You should check out their platform if you haven't. It is really super cool. A lot of inspiration for this post have been from using them.

This is one of my hobby project attempt at building just the Autoscaling feature for Hadoop1 clusters if had been part of say Qubole team back in 2012.

In this blog post I talk about the implementation goals and/or hows building this as part of InMobi's HackDay 2015 (if I get through the selection) or go ahead and build it anyways on that weekend.

For every cluster you would need the following configuration settings

• minNodes - Minimum # of TTs you would always want in the cluster.
• maxNodes - Maximum # of TTs that your cluster would like to use at any point in time.
• checkInterval - Time unit in seconds to check the cluster for compute demand (default - 60)
• hustlePeriod - Time unit in seconds to monitor the demand before we go ahead with upscaling / downscaling the cluster. - (default - 600)
• upscaleBurstRate - Rate at which you want to upscale based on the demand (default- 100%)
• downscaleBurstRate - Rate at which you want to downscale (default - 25%)
• mapsPerNode - # of map slots per TT  (default - based on the machine type)
• reducersPerNode - # of reduce slots per TT (default - based on the machine type)

Assumptions
- All the nodes in the cluster are of same type and imageId - easier during upscaling / downscaling.
- All TTs will have Datanodes also along with it

Broad Goals

- Less / No manual intervention at all - We're talking about hands-free scaling and not one click scaling.

- Should have less / no changes in the framework - If we start making forks of Hadoop1 / Hadoop2 to support certain features for autoscaling then most likely we'll have a version lock which is not a pretty thing 1-2 years down the lane. 

- Should be configurable - For users willing to dive deeper for configuring their autoscaling they should have options to do that. Roughly translates to being all blue configurations having sensible defaults.  

Larger vision is to see if we can make the entire thing modular enough to support any type of scaling. 

Please do share your thoughts if you have any on the subject. 

Online Virtual Reverse Auctioning

Well this is a small idea for an online event for a technical symposium at our college campus. Well here goes the details of it. Feel free to post your comments on this idea.

Name of the Event - Virtual Reverse Auctioning

Event Description

• Just like any auctions' website, this one will also hold auctions of various (virtual/random/nonsense) items. 
• Generally in any auctions, the bidder who bids the highest gets the item with the price paid, but in Reverse Auctions, the person with the lowest bid wins the bidding (at the end of the time). 
• To start with participant are given Rs. 100/- (negotiable) in their accounts to start off with. 
• To maintain order in the system against all participant posting Rs. 1 for all the items, I have come up with a rule for posting the bidding amount. 
• Least value of the bid can be only 95% of the current lowest bid value.
  Example - If the initial cost of the item is Rs. 100, and user A bids with Rs. 95/-, current cost of the item becomes now 95. User B, can now post a lowest of Rs. 90.25, highest being any value (no upper limits actually). 
• Each bidding costs the user a Buck. (Rupee. 1 /-) 
• Well again there is a small consideration to be made here, each bidding can be made as a pay-use or a bid on an item can be made as pay-use.
  Example - For every bid, I place on Item I can cost me Re. 1 /- or all the bids I place on an Item I can cost me Re. 1/-. I leave this for you to decide (post what you feel below as a comment)
• Upon winning the bid, the original cost of the item is credited to the user's account. Thus the user get more money to bid more on the system. 
• Since a limiting criteria of only 95% of the current lowest bid can be made, this keeps the participants checking in to the game often to see if anyone has made a lower bid than him/her. 

Winning Criteria - Participants with the highest cash on hand (plus number of items traded on the market, well this is again up to you. Feel free to post what you feel below as a comment) wins the game. 

General Doubts 

1. Organizer of the event is given the Admin access to add items to the system. He can set the Name, type, initial cost, time period of auction of all the items currently enabled in the system. More about this to the developing team. 
2. All the items will be completely virtual. It can range from a banana to iPhone - you are limited by your imagination. 
3. System is yet to be developed. Hoping to create a team (of max 4) to build this system. Its fairly simple to do so. I would like to see second and third years volunteer in this (first years are also welcome) to make it a reality. 
4. If you are interested in developing this idea, contact me, I will pour in more ideas.

Disclaimer: This Game plan is completely a creative idea, from Ashwanth Kumar. This was shared with a sole intention that it might be useful. If you already have a similar or the same idea, I would love to discuss about your implementation idea.