RootConf 2016

Last week I attended Rootconf 2016. My first of RootConfs and second HasGeek event. The event was organised good compared to Fifth Elephant in 2014. You can find my personal notes from various talks at https://github.com/ashwanthkumar/rootconf-2016.

At 4PM on Day 2, I got an opportunity to go to Apigee's Bangalore office to present about Matsya. I prepared a presentation hoping to talk at one of the Flash Talks but never got a chance at the event, but it did come useful during ApiGee's session :)

How not to use awk

Today I was working on automating AMI creation for a project and ended up using packer. So packer has this thing called -machine-readable output which can be easily parsed as CSV (at the very least). I ended up writing a bash script to parse the output.
I started off with using while loop + awk for parsing (in that order) and emitting the lines that contains the artifact information. As you can see above for parsing around 1700+ lines it took > 6 seconds.



Afterwards I refactored the code to use awk first and feed that output to while loop, which actually ran 100x faster.


I didn't knew AWK was so good at processing things at scale (if I may). 

Building GoLang on SnapCI

I've been a long time user of SnapCI. I recently started playing around with GoLang. I started writing tests and releases on some of the hobby projects and it's high time I start setting up a CI for it. Unfortunately SnapCI doesn't support GoLang out of the box and making it work is a bit of a hassle. After fighting with it for more than a hour, I figured out how to compile my go project.

Disclaimer - My project on my local machine is part of my custom $GOPATH so all the `go` tool chain works. Makefile also assumes this. If you're used to building your project on a non-standard directory structure, this is not for you.

Thanks to this post talks about how to setup GoLang on SnapCI. It doesn't work anymore out of the box, but does have the fundamental things to get started. Final settings that worked for me

Hope this helps.



Apart from the above, we also need to setup the GOPATH, in order for go to work. On the stage configuration page there is a section called Environment Variables for this stage in there we need to set the name to GOPATH, and the value to /var/snap-ci/.

Find that missing Host in Hadoop Cluster

I've started to manage 200 node hadoop clusters recently at work. All these are running on AWS with latest CDH5. We went from separate HDFS + TT model to co-locating TT and DN daemons together.

These machines are all Spot machines backed by a ASG (Auto Scaling Group). If any of them die because of spot prices, they would come back up in a while. So to manage these machines better we attach our own custom generated DNS names to these machines.

Once in a while, the machines that come up doesn't have either a TT or DN daemons running. They would have failed at startup for variety of reasons. The task was to find that missing hosts (generally 1 or 2) of the lot. So I wrote a script that would help us get the missing hosts which don't run one of the process.

Gist - https://gist.github.com/ashwanthkumar/3624a4e69ab26236a746

Kitchen Bento Setup

After struggling couple of days trying to create a new virtualbox with some packages pre-installed for my kitchen tests, I ended up frustrated only to later figure out about mitchellh/vagrant#5492. So, I ended up creating kitchen-bento-setup which would help me create new virtual boxes from the base opscode box.

For reasons unknown to me, I don't get "SSH Authentication Failure" when packaging the VBox using `vagrant package` the first time from base opscode box, any consequtive `vagrant packages` from that pre-built box causes the failure. 

ClassNotFound inside a Task on Spark >= 1.3.0

Context - Spark 1.3.0, Custom InputFormat and InputSplit.

Problem - At my work, I had a custom InputSplit definition which had another class A object which then I need to pass it to my Key. I then have a Spark job that reads using my custom InputFormat and things were all fine on Spark 1.2.0. When we upgraded to 1.3.0 things started breaking with the following stack trace.

Caused by: java.lang.ClassNotFoundException: x.y.z.A
 at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
 at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
 at java.security.AccessController.doPrivileged(Native Method)
 at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
 at java.lang.Class.forName0(Native Method)
 at java.lang.Class.forName(Class.java:270)
 at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:625)
 at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612)
 at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
 at org.apache.spark.util.Utils$.deserialize(Utils.scala:80)
 at org.apache.spark.util.Utils.deserialize(Utils.scala)
 at x.y.z.CustomRecordSplit.readFields(CustomRecordSplit.java:91)

Solution - It took me a while to realize that I've been using Spark's Util object to serialize and deserialize the object (x.y.z.A). The fix was very simple

objA = Utils.deserialize(buffer, Utils.getContextOrSparkClassLoader());

Looks like in the earlier versions the __app__.jar is getting added as part of the Executor and Task classloader but not in the latest versions. When I passed the Context ClassLoader to the deserialization it worked perfectly fine. 

Lessons

- Don't use Spark's Util method. Even though the Util is a private[spark] object since I'm accessing it from a Java class the scala package access protection doesn't seem to apply. I never knew that until now. 

- Always use Utils.getContextOrSparkClassLoader() when doing Java Deserialization in Spark.

Parser Combinator

After fiddling with RegexParser for a while in Scala, I realized how much I missed learning Automata properly at college. I was migrating a 140-line REGEX from MySQL to Scala at my work and learned a lot of new things in the process. It was during one of those times one my mentors - yellowflash, helped me understand some of the forgotten concepts about left factoring, recursive grammars, etc.

In the process he were discussing about how would I go about writing RegexParser if I had to write it by hand myself? The exercise was to help me understand how the Parser Combinator works which would help me write better grammar. We did some scribbling on the paper and decided to implement it in Scala. You can find it on https://github.com/ashwanthkumar/parser-combinator. It is just a start, still a long way to go. Looking forward to it, it should be fun.

Slack Java Webhook

GoCD recently added support for notification extension point. I've started building slack notification plugin (its a WIP here). As part of that I wrote a Java client for Slack Webhooks. Although I found a java library here, which said it was published but I couldn't find it anywhere on sonatype / maven central. I can't even publish it, so I took that as an inspiration and wrote my own implementation on https://github.com/ashwanthkumar/slack-java-webhook.

Usage

new Slack(webhookUrl)
    .icon(":smiling_imp:") // Ref - http://www.emoji-cheat-sheet.com/
    .sendToUser("slackbot")
    .displayName("slack-java-client")
    .push(new SlackMessage("Text from my ").bold("Slack-Java-Client"));

It gets posted in the slack channel like below

Dependencies

For Maven,

<dependency>
  <groupId>in.ashwanthkumar</groupId>
  <artifactId>slack-java-webhook</artifactId>
  <version>0.0.3</version>
</dependency>

For SBT,

libraryDependencies += "in.ashwanthkumar" % "slack-java-webhook" % "0.0.3"

Thrift Notes on Default values

I have been using Apache Thrift at production for close to 2 years now and something that we did not know for a long time was how to assign default values to thrift's collection types like map, list, set.

It so seems that Thrift IDL default value assignments can be JSON style values.