RootConf 2016

Last week I attended Rootconf 2016. My first of RootConfs and second HasGeek event. The event was organised good compared to Fifth Elephant in 2014. You can find my personal notes from various talks at https://github.com/ashwanthkumar/rootconf-2016.

At 4PM on Day 2, I got an opportunity to go to Apigee's Bangalore office to present about Matsya. I prepared a presentation hoping to talk at one of the Flash Talks but never got a chance at the event, but it did come useful during ApiGee's session :)

Marathon-Alerts: Alerting tool for Marathon Apps

Another month has passed by, and a bunch of open source contributions that has happened. I recently published on a blog post on Indix's blog titled - "Marathon-Alerts: A Tool for Keeping Your Apps on Marathon Under Radar". It is an official post for open sourcing the marathon-alerts tool to the outside world. Without further delay, here's the tl;dr version of the post.

Marathon-Alerts, is a tool to monitor apps running on marathon. Marathon while reaching it's 1.0.0 milestone (as of writing), it still doesn't have first class support for alerts for tasks running via it. If you're on all containers, then there are lot of tools out there in the market which can help you monitoring and alerting (Sysdig for example).

Unlike k8s, marathon doesn't limit to running only containers but arbitrary commands too. This is where specialised tools like marathon-alerts comes into play. Today we've notifier only for Slack, but it's quite easy to extend them to other sinks which can integrate to your existing infrastructure.

Above screenshot is an example alert that comes on our slack channels when one of our apps went down because of an error. 

This is one of my best open source contributions in recent times. Check out the project on https://github.com/ashwanthkumar/marathon-alerts, it's released under Apache 2 License. Fork away!

How not to use awk

Today I was working on automating AMI creation for a project and ended up using packer. So packer has this thing called -machine-readable output which can be easily parsed as CSV (at the very least). I ended up writing a bash script to parse the output.
I started off with using while loop + awk for parsing (in that order) and emitting the lines that contains the artifact information. As you can see above for parsing around 1700+ lines it took > 6 seconds.



Afterwards I refactored the code to use awk first and feed that output to while loop, which actually ran 100x faster.


I didn't knew AWK was so good at processing things at scale (if I may). 

Introducing Matsya

I recently wrote a blog post on Indix's Engineering blog about Matsya. Do check it out at http://www.indix.com/blog/engineering/matsya-auto-scaling-availability-zone/. I'm going to summarise the blog post here in TLDR; version.

We have 100% of our Hadoop workloads running on AWS Spot infrastructure. One of the prominent issues we've have faced is "Spot Outages". In order to solve that, we went cross AZs. This in return resulted in huge data transfer costs especially for systems like Hadoop HDFS. We realised the only way to survive this madness is to be on a single AZ but intelligently move to another AZ when there's a Spot spike in the current one. That worked :) There were cases when we couldn't find any AZ within your bid budget. Typically our bid budget was 100% of OD price. During such times, it can fallback to OD instances until the surge ceases.

I wrote a tool to automate the above process, and that's Matsya. Matsya is the name of first incarnation of Lord Vishnu in this world. The main part of the story is he takes a form of a Fish (with a horn) and carries the Seven Sages (saptarishis) through safety on the Judgement day. The name was so apt since Matsya is able to carry the cluster from one Spot Market to another (read cheapest) during a Spot surge.

Bonus - Slides to my presentation I gave at Chennai Devops meetup on January 2016 can be found at http://j.mp/to-matsya. 

Winning #GoPluginChallenge

Winning the #GoPluginChallenge was a very big deal for me. Why? Being acknowledged by a team that rejected me in an interview 3 years ago gives at most satisfaction of some kind of achievement in life. All thanks and credits goes to my mentors - Rajesh and Sriram. Special thanks to Manoj who gave me all the motivation for building the Github PR plugin. It is also really nice to know both the plugins that I submitted have won together.

GoCD - Slack Build Notifier

In my last post I wrote about a GoCD plugin that I've been working on. I finally got to complete it this weekend. Check it out https://github.com/ashwanthkumar/gocd-slack-build-notifier.  This is how the final result looks like

There are two features in the plugin that I'm really happy about (apart from pushing messages to slack)

1. Pipeline rules set. It is heavily inspired from the current email notification framework available as part of GoCD. Check out the "Pipeline Rules" section in README.
2. Notifier is pluggable. Slack Notifier is provided out of the box. With very little change, one can write any type of notifications transport using the existing framework. 

Overall it was a time well spent that helped me to write (which I guess is) first notification plugin in the GoCD community.

Java Utils Library

After a long time I seem to be writing Java code more often recently. A bunch GoCD plugins, code kata sessions with friends and things like that. I saw there are few things like List transformations, filter, I automatically start searching for Option / Some and None implementations. Simple solutions would be just write it in Scala, right? I know, but there are places where I wasn't. Example was GoCD plugins. Reasons being - Scala standard library is heavy and usually causes OOM on Agent without increasing heap sizes and final jar is also heavy in terms of size.

Check out https://github.com/ashwanthkumar/my-java-utils. If you find some implementations not so efficient or can be done better, please do let me know.

Features

List

• Lists#map
• Lists#filter
• Lists#foldL
• Lists#find
• Lists#isEmpty
• Lists#nonEmpty
• Lists#mkString

Set

• Sets#copy
• Sets#isEmpty
• Sets#nonEmpty

Iterable

• Iterables#exists
• Iterables#forall

Lang

• Option / Some / None
• Tuple2 / Tuple3
• Function
• Predicate

Dependencies

For Maven,

<dependency>
  <groupId>in.ashwanthkumar</groupId>
  <artifactId>my-java-utils</artifactId>
  <version>0.0.2</version>
</dependency>

For SBT,

libraryDependencies += "in.ashwanthkumar" % "my-java-utils" % "0.0.2"

JSFoo Chennai Runup Event

Interesting times it has been a last few months. One of 'em was JSFoo Chennai Runup event at Indix office.

I wrote a blog post (my first official blog post, yay!) on our blog that summaries the event.

Our Chennai Office Hosted a Run-up Event for @hasgeek @jsfooindia 2014. Read About it on Our Blog. @_ashwanthkumar http://t.co/l1AfuvdQ9U

— Indix (@Indix) September 11, 2014

Thrift Notes on Default values

I have been using Apache Thrift at production for close to 2 years now and something that we did not know for a long time was how to assign default values to thrift's collection types like map, list, set.

It so seems that Thrift IDL default value assignments can be JSON style values.

BlueIgnis - Feature Screencast - Closed Beta Preview

UPDATE on 2nd May, 2012 IST - BlueIgnis Closed Private Beta is closed. Thanks to everyone who tested it.

An year old dream, 100's of prototypes, more than10 months of designing, finally everything getting shape. BlueIgnis is finally launched for Closed Private Beta. Quite some ppl are giving it a spin right now. Just love to see the system handle the pressure and scale as needed. Only thing is I am hard pressed on, are the server resources and though the system is horizontally scalable, I wish I can scale along with it to support more concurrent users and remove the limitations on the system. Just ranting about the status quo on the system.

Good thing is that system is a bit well tested with real users and it is all set for the demo happening today in about 5 hours from now. Though with the help of Salai I have completed some more new features to the system, I am not going to push them in until after the demo.

I just made a small 6 minute screencast on features of BlueIgnis, and I thought I will put it forward for you. Though the system is up and running, I would really appreciate if you do not create more than 1 account per user. The system has a hard limit on 50 users and it will lock the registrations immediately after that.

Apart from that, I have imposed the limits on Number of Campaigns per user and Time period for which Campaigns will be running since it starts. I will add another screencast once I am complete with integration of the new set of features.

You can find the screencast on http://youtu.be/VYJGQreoYbg

PS: I totally understand if you feel the Screencast is not of the mark, I have never recorded by own voice before. I will improvise on that. 

Introducing Scraphp - Web Crawler in PHP

Scraphp (say Scraph, last p is silent) is a web crawling program. It is basically built to be a standalone executable which can crawl websites and store extract useful content out of it. I created this script for a challenge posted by Indix on Jan 2012, where in I was asked to crawl AGMarket (http://agmarknet.nic.in/) to get the prices of all the products, and store their prices. I also had to version the prices such that it should persist across dates.

Scraph was inspired from a similar project called Scrappy, written in Python. This is not an attempt to port it, but just wanted to see how much similar properties can I build from it in less than a day.

One of the major features I would like to call it is, When you crawl the page you can extract entites out of it based on XPath. So basically when we crawl a page I create a bean whose properties are set of values got by applying the given XPath on the page. Each XPath is completely independent of the other. Currently Scraph supports creating only 1 type of object per page.

Hack into the source code, its well commented and easy to modify as per requirement. All the details of the crawling page, XPath queries are all provided in the configuration.php or you can supply your own config file, see the Usage. 

Code is available on my Git Repo - https://github.com/ashwanthkumar/scraphp 

I have tired my best to document the entire code well, and if you feel like any improvements can be made or you have got any suggestions? Please do not hesitate to fork and send me a pull request. 

I am Alive!

Been like ages since I wrote a post here. Lot of interesting things been going on around my life. Will try to keep this space updated in the forth coming weeks.

Joined in Mu Sigma as Intern last month and got into their Innovation and Development (generally R&D) department with Big Data. Work so far has been excellent, Playing with clusters, learning new languages, and some boring math, but still its all part of the game isn't?

PS: This is just an update post, to denote I am still alive and not dead. 

ARO - Doc Comment Parsing in PHP

Following the work on IoC (Inversion of Control), here I am again to present the second module of my project, ARO (At the Rate Of). ARO is a PHP Doc Comment parsing library useful for parsing the Annotations, descriptions, etc. from the Classes, Methods or Properties.

ARO uses PHP Reflection to get its job done. Well this library is again flexible and hence I thought I can release it as open source. Its hosted on GitHub (https://github.com/ashwanthkumar/aro-php). Feel free to fork or report an issue.

I tired my best to document it on the GitHub, so let me skip further details here.

Disclaimer: Reason I created this ARO is for using IoC effectively in the first place. Also there are many more advanced Doc Comment parsers in PHP. This module is just what I came up in around 5 hours.

IoC-PHP - My (little) Inversion of Control on PHP

After around 5 hours, I am happy to say I have completed my first module in my BlueIgnis project. It is the IoC (Inversion of Control) module. Now, I can dynamically Inject dependencies in my application as and when needed.

This is definetly not the first IoC implementation in PHP, but I just wanted to create one for my needs. After all its my project and I would like to code it from scratch (I can't believe I am saying this).

A note of caution: I am a newbie to the entire IoC thingy, I know very little or I just misunderstood the entire concept. So, I wanted to learn from the community. Hence I decided to release it as Open Source code so that people can have a look at it or may be even improve it further.

I currently don't bother much with the Licensing issues. If you like the code, or you think it can be improved further, please let me know.

For people who does not know what IoC is, have a look here.

Next step would be to achieve the Design Goals in IoC using Annotations support.

Link for the project - https://github.com/ashwanthkumar/ioc-php

BlueIgnis - Starting Finally

Finally after almost 8 months of planning, modelling, and designing process - BlueIgnis is finally taking place. Good part is that my mentor has asked me to do Oracle ThinkQuest as the final year project. I was hunting for a good topic to build. The category I am eligible is "Application Development", and when I was wondering I decided its time I spend some time for BlueIgnis (aka Social Heat).

Taking the wonderful experience from building a custom framework using simple design patterns (more on this in a later post) for Webnaplo, and from the previous TGMC project (Back To My Village) - Reformists; Past 2 days went good building the BlueIgnis architecture with more new features which I only used to dream before.

I am all excited to work on this. This time with more features packed into its design. Since I am planning to release this as a standalone app and not as SAAS (I wish I can do it), it runs on your infrastructure.

For people who does not know what BlueIgnis is, you can refer my earlier blog post which gives a gentle introduction about its functionality.

Keep watching this space for more information and updates regarding BlueIgnis.

Got Placed!

Title says it all! I got placed and yeah, its been around 2 months, and there were many interesting things that went on actually. I'm not sure, If I'll cover them all up, but will try to do it.


I got placed in Mu Sigma (http://www.mu-sigma.com/) as Business Analyst. Yeah, yeah I know; its not an IT company. Its a business oriented company. You see that actually gives me all the more reason to join here. These people come up with the solutions to the business problems (now, do not ask me yet for meaning of business problems) for their customers. Which goes without saying the amount of data they will be processing, sure require some deep data mining techniques to get it done.


Guess what more? MMBUZZ (http://www.mu-sigma.com/dothemath/id_mtex.html), this particular project got me the job in the first place. Social Heat aka Blue Ignis project of mine does the same work! Vice president of the company was impressed and yeah, he offered me a job. I got a laptop bag, along with my offer letter. All the more reason to join. :-P

I've also been on the hunt for some internship programs lately, got a couple of offers. Yet to decide which one to take.

Final semester, with hardly less than a week of working days left in my college life; What have I learnt? What does it mean?

(After almost 1 hour of thinking)

Aahh.. I'll put that up later not right here.

Happy thing, I got placed. Thanks to SASTRA and My Mentor. 

Fo2Hub - Launching Soon

Yeah you got it right. FotoHub has become Fo2Hub. Reason? Simple, I was not able to get a domain name as fotohub.com, so I got one for fo2hub.com. A million thanks to my bro, who got me a new domain and hosting. I'm now under no restrictions, and no problems. Thanks a lot bro!

Fo2Hub - Photo Sharing for Social Nerds, will be launching soon. I've currently put up a small landing page on the domain. Since, my main domain ashwanthkumar.in is also being moved. It might be currently unavailable for max. 24 - 72 hours. You can still access a read only archived copy at here, This location will not be updated anymore.

Fo2Hub.com - Launching Soon!

As always, any feedback or suggestions are always welcome.

FotoHub - Photo Sharing for Social Nerds

Update 10/05/2011 - FotoHub is now Fo2Hub. You can find more details of it here.

I was just updating my profile on my website, when I realized it would be so nice to add a personal gallery. Flickr was the obvious choice. I logged into my Flickr account to realize only that I've not logged in since my 11th grade. He.. He.. So, what do I know? Picasa is already available on My Google Profile, but I've many of my photos on my FB too. What about the one's that I am tagged in? Oh My God! Maintaing a profile is really a pain in the ass. Grrr..!! That does it.

I'm going to start my own photo sharing web application. One, that can help synchronize from Facebook, Picasa, Flickr (and many more). An application in which probably I can set to synchronize specific albums, from my social presence online. It basically targets people with a very large social presence (not exactly like me, but ... you get it right?)

Welcome a new baby to AshwanthKumar.in Labs, FotoHub - A Photo sharing space, for social nerds.

"Instead of maintaining a separate accounts for the purpose, why not create an album syndicating pictures from multiple sources (probably with your tags, which will be added shortly after release) and accessible via an easy to use API?", the thought that got me designing FotoHub for public.

Though Social Heat is going on good, and due release for (pre-beta) public testing very soon. FotoHub seems to be a good stress reliever for me. Watch the space for more updates on FotoHub.

PS: Still I'm keeping my eyes and ears open, for people who can still suggest me a web application (not android app or desktop picture viewer) that can do this job for me.

iBlue Semantic Extractor - Now Speaks 52 Languages

Another Good morning to you people. Today, I've an interesting update regarding iBlue Semantic Extractor. Semantic Extractor component was initially build as a project prototype for Mr. Guruprasad S. Srivatsav back last year. It simply extracted entities out of the text, find their relations, and categories. When it was first created, it had a 1000 word limit, and English language only detection.

Today, I present you iBlue Semantic Extractor with 52 language support. You can enter the text in any of the 52 languages, and it is translated to English dynamically and then identifies the entities, relations and categories.

52 Languages supported are: Afrikaans, Albanian, Arabic, Belarusian, Bulgarian, Catalan, Chinese Simplified, Chinese Traditional, Croatian, Czech, Danish, Dutch, English, Estonian, Filipino, Finnish, French, Galician, German, Greek, Haitian Creole, Hebrew, Hindi, Hungarian, Icelandic, Indonesian, Irish, Italian, Japanese, Latvian, Lithuanian, Macedonian, Malay, Maltese, Norwegian, Persian, Polish, Portuguese, Romanian, Russian, Serbian, Slovak, Slovenian, Spanish, Swahili, Swedish, Thai, Turkish, Ukrainian, Vietnamese, Welsh, Yiddish

You can visit iBlue Semantic Extractor Component for free preview here - http://ashwanthkumar.in/labs/iblue/

Next Milestone - Identify the entities natively from the given language.

Project UIM - Going Good

Update May 3rd, 2011 - Now UIM consumes Blogger feeds too. :)

For over a week or is two? I don't seem to remember. I started working on something, I would like to call it as Project UIM (Unstructured Information Management). Its basically iBlue - mini, something like that. Trying to maintain a repo of largest Structured Information for computer systems.

I took the following site as my source of content - Gizmodo, LifeHacker, Mashable and Tech Crunch. For almost a week all their articles posted, within an hour comes into my system for processing. Using the Semantic Extractor Component (available here for preview), I was able to extract some useful re-usable information (yeah! re-usable information) out of them.

I just ran some diagnosis on them today, and would like to share some stats regarding that. My system has consumed over a 1061 (at the time of writing this space) articles from the above said sources, and identified over 3368 entities (in over 34 types), 18 categories and 46 relations.

Below a stats of the entity count and its type. :-)

count	type
124	City
522	Company
5	Continent
64	Country
6	Currency
4	EmailAddress
7	EntertainmentAwardEvent
115	Facility
11	Holiday
852	IndustryTerm
2	MarketIndex
10	MedicalCondition
24	Movie
19	MusicAlbum
29	MusicGroup
18	NaturalFeature
15	OperatingSystem
154	Organization
617	Person
13	PhoneNumber
1	PoliticalEvent
380	Position
80	Product
11	ProgrammingLanguage
39	ProvinceOrState
49	PublishedMedium
3	RadioStation
8	Region
10	SportsEvent
5	SportsLeague
114	Technology
16	TVShow
1	TVStation
28	URL

That's in my opinion a very decent for a pre-alpha implemented system.

I'm planning to expand the data source, to try to see how well Project UIM can tame the beast, Internet. Any other suggestions regarding the same is welcome. Please leave them as a comment.