BlueIgnis - Feature Screencast - Closed Beta Preview

UPDATE on 2nd May, 2012 IST - BlueIgnis Closed Private Beta is closed. Thanks to everyone who tested it.

An year old dream, 100's of prototypes, more than10 months of designing, finally everything getting shape. BlueIgnis is finally launched for Closed Private Beta. Quite some ppl are giving it a spin right now. Just love to see the system handle the pressure and scale as needed. Only thing is I am hard pressed on, are the server resources and though the system is horizontally scalable, I wish I can scale along with it to support more concurrent users and remove the limitations on the system. Just ranting about the status quo on the system.

Good thing is that system is a bit well tested with real users and it is all set for the demo happening today in about 5 hours from now. Though with the help of Salai I have completed some more new features to the system, I am not going to push them in until after the demo.

I just made a small 6 minute screencast on features of BlueIgnis, and I thought I will put it forward for you. Though the system is up and running, I would really appreciate if you do not create more than 1 account per user. The system has a hard limit on 50 users and it will lock the registrations immediately after that.

Apart from that, I have imposed the limits on Number of Campaigns per user and Time period for which Campaigns will be running since it starts. I will add another screencast once I am complete with integration of the new set of features.

You can find the screencast on http://youtu.be/VYJGQreoYbg

PS: I totally understand if you feel the Screencast is not of the mark, I have never recorded by own voice before. I will improvise on that. 

BlueIgnis In Making - Pre Alpha Preview

I wonder how many have you experienced what I had for the past 2 days. I had a phone call from my project co-ordinator stating that I might not get my degree. There has been a lot of mis-communication by the person involved in the issue. Shit Happens!! Now I have my 2nd and 3rd Review on Monday, I consider myself special, you know ;-)

I never really started to build something for my project until Thursday when I had the phone calls. I shifted all my engagements at work till Tuesday to get into the damn thing.

Scenario - I had 2 days for building a demo, 1 day for preparing the documentation, and 1 day for travel (which I generally do at night to save time).

To my own astonishment I was able to come with a demo-able something in 1 day, which I shared with some colleagues of mine to improvise on demand and fix some bugs. After around 1.5 days of work, here are some of the screen shots from the application.

Ingredients^# - PHP (as always) + MySQL + Redis + Twitter Bootstrap + jQuery +
Coke + Pressure to get a Degree (*wink*)

# Prioritized on usage

First the Dashboard, it shows you the overall stats of the Twitter feeds and their sentiment division among Positive, Negative and Neutral. Currently I have implemented only Twitter feeds, Facebook and G+ should not take much time to implement.

Next is the Key Influencers. People who shape the conversation about your product / event / campaign. It is shown in a network form, where the center node is a picture that represents your campaign, and all other nodes represent the users. User's contribution towards the campaign denotes the higher size of the node. One of my friends suggested to improvise on this using Klout score too, seems like a very nice idea, need to see if it is possible and it would be a Kicking Ass feature if implemented.

Next comes the Realtime Twitter Dashboard. As we get in Tweets to the system, we push them to the screen so that users see what people are talking about their campaign in real time. I do not know UX or anything related to UI, but I am a freaking good user.

I know the problems user's might face and one such problem was. When I kept on adding tweets to the page in realtime, for a high frequency campaign. After around 10 - 15 min, my browser started becoming slow / non-responsive. Too much of content on page was not at all a good idea, hence now we start off with 10 tweets pre-loaded into the page and it gets updated in real time. Still there are no more than 100 tweets at the page (on either side) at any given point of time. Having the browser and page more responsive.

I am still open for any further improvements / changes by an expert. 

Next comes the Realtime Sentiment Dashboard  - Real time feed of sentiments for the campaign. As we process the tweets as Positive / Negative they drop by into the screen in real time.

Apart from this we have a login screen and a create campaign page which helps you create campaigns. Once I am done with this documentation I really wish to spend some considerable amount of time on this to add more features and analytics to it.

Would you be interested on using it? Do you think there are additional features that you might be interested in seeing in this? UX / UI tips for the novice here? Any suggestions are always welcome and drop by a comment stating what you feel about it. 

Twitter Streaming Limit Workaround

I was working on my final year project (BlueIgnis) which uses Streaming Twitter API. I had the following understandings from them (on free version):

1. One Account can open only One Streaming Connection at any given time
2. One IP may be associated with only One Account while streaming. Rotation of Streaming connections based on multiple accounts are not allowed. May lead to IP Ban. (All the more reason to use EC2 Instances for Streaming :P)
3. One Streaming connection may allow upto 400 tracks (different keywords) to filter from.
4. Reply to 402 Error codes with proper HTTP Status.
5. Should use non-aggressive re-connect policies, must give substantial amount of time in-between subsequent requests.
6. Periodically we must stop the Streaming Connection, add more tracks (keywords) to the list and re-start the connection, rather than individual connections for multiple times.

Based on these understandings, I came up with own Architecture for Twitter Streaming. Below diagram represents the overall architecture of my application with respect to Twitter Streaming Component.

Hosting the Twitter Streaming on an EC2 Instance, we can achieve 400 tracks (keywords) per node which can handle approx. 30 - 50 customers based on my use-case. I periodically (~10 min) check if there are any new tracks that needs to be added to the node until it becomes 400. Since I need to know which user requested the track, which is not possible to get from the current way the Streaming API works. 

So I decided to build a Local Firehouse, where in I stack all the tweets for all the tracks, all in a single location. Then, I use a FullText Search feature of MySQL (my datastore) to search for the related tweets continously so that I can achieve the feel of a bit delayed streaming yet close-to-realtime processing.

If you have any better ways to get things done, please let me know.

Consuming CommonSense Knowledge on MR

ConceptNet5 one of my all time favorite dataset available out there. I am working with it in more detail for BlueIgnis (more details on this later). After coming into Big Data of Mu Sigma, this thought has been lingering in my mind. CN5 is really a large dataset ~24 GB of exported JSON data, goes upto ~111GB with indices (as explained on the link), y not use MapReduce to spice things up a bit?

When I joined the company last month, I was told to start with R (Statistical Language). I always wanted to port the Divisi2 to Java or PHP so that I can hack into it more. After a day of getting to know R, I wrote a simple wrapper in R to build CommonSense Matrix. Not the entire thing, just a sample of it with imaginative data and made it work (R code here).

Well basically its all doing SVD and operating on its components - U, V and E (Sigma), to get make predictions. Blah.. blah.. you could have read it in the page in detail if you can understand math (unlike me).

What am I trying to do here?
What I was wondering is, Number of Concepts (nodes) in CN5 exceeds way more than what I can imagine (I am yet to count them as I still have 32-bit system with me, and MongoDB can't hold more than 2GB of data on Win32 systems. Sigh!), not to mention the relations of each concept the same with the columns of the matrix. If only I could transfer data from Mongo to HBase and use Mahout's SVD Implemenation to build the required matrice and store it in HBase (again). I guess that should put me to use commonsense dataset based processing of data. I need to process realtime Tweets and FB Posts in Storm for BlueIgnis, would it match the performance on Real-time basis? Is this even possible? I don't have answers to these and many related questions yet. Just an idea, yet to hack into it more.

Let me know if you have already implemented this or working on similar road.

Updates:
Some interesting thought on GraphLab usage and performance over Mahout's implementation, here. (See the comments).

PS: Above idea was thought over a cup of tea and some cake at hand with no work to do. If you have already got anything like this, I would love to hear from you. 

ARO - Doc Comment Parsing in PHP

Following the work on IoC (Inversion of Control), here I am again to present the second module of my project, ARO (At the Rate Of). ARO is a PHP Doc Comment parsing library useful for parsing the Annotations, descriptions, etc. from the Classes, Methods or Properties.

ARO uses PHP Reflection to get its job done. Well this library is again flexible and hence I thought I can release it as open source. Its hosted on GitHub (https://github.com/ashwanthkumar/aro-php). Feel free to fork or report an issue.

I tired my best to document it on the GitHub, so let me skip further details here.

Disclaimer: Reason I created this ARO is for using IoC effectively in the first place. Also there are many more advanced Doc Comment parsers in PHP. This module is just what I came up in around 5 hours.

IoC-PHP - My (little) Inversion of Control on PHP

After around 5 hours, I am happy to say I have completed my first module in my BlueIgnis project. It is the IoC (Inversion of Control) module. Now, I can dynamically Inject dependencies in my application as and when needed.

This is definetly not the first IoC implementation in PHP, but I just wanted to create one for my needs. After all its my project and I would like to code it from scratch (I can't believe I am saying this).

A note of caution: I am a newbie to the entire IoC thingy, I know very little or I just misunderstood the entire concept. So, I wanted to learn from the community. Hence I decided to release it as Open Source code so that people can have a look at it or may be even improve it further.

I currently don't bother much with the Licensing issues. If you like the code, or you think it can be improved further, please let me know.

For people who does not know what IoC is, have a look here.

Next step would be to achieve the Design Goals in IoC using Annotations support.

Link for the project - https://github.com/ashwanthkumar/ioc-php