RootConf 2016

Last week I attended Rootconf 2016. My first of RootConfs and second HasGeek event. The event was organised good compared to Fifth Elephant in 2014. You can find my personal notes from various talks at https://github.com/ashwanthkumar/rootconf-2016.

At 4PM on Day 2, I got an opportunity to go to Apigee's Bangalore office to present about Matsya. I prepared a presentation hoping to talk at one of the Flash Talks but never got a chance at the event, but it did come useful during ApiGee's session :)

Marathon-Alerts: Alerting tool for Marathon Apps

Another month has passed by, and a bunch of open source contributions that has happened. I recently published on a blog post on Indix's blog titled - "Marathon-Alerts: A Tool for Keeping Your Apps on Marathon Under Radar". It is an official post for open sourcing the marathon-alerts tool to the outside world. Without further delay, here's the tl;dr version of the post.

Marathon-Alerts, is a tool to monitor apps running on marathon. Marathon while reaching it's 1.0.0 milestone (as of writing), it still doesn't have first class support for alerts for tasks running via it. If you're on all containers, then there are lot of tools out there in the market which can help you monitoring and alerting (Sysdig for example).

Unlike k8s, marathon doesn't limit to running only containers but arbitrary commands too. This is where specialised tools like marathon-alerts comes into play. Today we've notifier only for Slack, but it's quite easy to extend them to other sinks which can integrate to your existing infrastructure.

Above screenshot is an example alert that comes on our slack channels when one of our apps went down because of an error. 

This is one of my best open source contributions in recent times. Check out the project on https://github.com/ashwanthkumar/marathon-alerts, it's released under Apache 2 License. Fork away!

Introducing Matsya

I recently wrote a blog post on Indix's Engineering blog about Matsya. Do check it out at http://www.indix.com/blog/engineering/matsya-auto-scaling-availability-zone/. I'm going to summarise the blog post here in TLDR; version.

We have 100% of our Hadoop workloads running on AWS Spot infrastructure. One of the prominent issues we've have faced is "Spot Outages". In order to solve that, we went cross AZs. This in return resulted in huge data transfer costs especially for systems like Hadoop HDFS. We realised the only way to survive this madness is to be on a single AZ but intelligently move to another AZ when there's a Spot spike in the current one. That worked :) There were cases when we couldn't find any AZ within your bid budget. Typically our bid budget was 100% of OD price. During such times, it can fallback to OD instances until the surge ceases.

I wrote a tool to automate the above process, and that's Matsya. Matsya is the name of first incarnation of Lord Vishnu in this world. The main part of the story is he takes a form of a Fish (with a horn) and carries the Seven Sages (saptarishis) through safety on the Judgement day. The name was so apt since Matsya is able to carry the cluster from one Spot Market to another (read cheapest) during a Spot surge.

Bonus - Slides to my presentation I gave at Chennai Devops meetup on January 2016 can be found at http://j.mp/to-matsya. 

Using AWS from Corporate Firewall

I know the title is somewhat pretty straight. I just wanted to share one of my learnings when I tried to AWS services from within Corporate Firewall.

Rule of the Thumb - Never add all the IPs to your ruleset on your firewall that Amazon releases on its forum.

Amazon Web Services (AWS) has a concept called "Elastic IPs". This allows you get a set of Static IPs and then use them with EC2 instances or VPC service. Try to create a bunch of IPs around 10 - 15 (depending on your purpose) and assign them manually to the instances as and when you create them.

Elastic IPs are not free, but the cost involved in holding them even when you don't use them is very less. So its worth the bargain, especially if you belong to an organization that harps on data security.

PS: I am no way related to Amazon, but I just love their services :-)