Tweetoem - Discovering Art in Tweets

Title is slightly mis-leading, but I cannot think of a better way to put it. Let me tell you the story behind building of "Tweetoem" (Tweets + Poem). 

Just in case you stumbled on this post first, Tweetoem is live on http://labs.ashwanthkumar.in/tweetoem/

Tweetoem is highly inspired from 140verses (http://140verses.com/). One of my friends shared this link (he is one of the developer of 140verses) on his facebook timeline. It was love at first sight for me, when my reverse-engineering brain cells got activated. I looked into the tweet generating poems, I loved the idea, I loved the execution of work. It has been quite some time until I did any hacks (none after I started working, life of intern seems a bliss now). So to put my spirits high and to brush up on my PHP skills I wondered if I can put something like that in < 6 hours of work? And "Tweetoem" was the result of that work. 

Algorithm (working behind the scenes)

1. Get the tweets from Twitter
2. Strip user_mentions, hashtags, and links of the tweets
3. Get the last word and reverse it
4. Calculate the Metaphone value of that word
5. Store the Tweet + Metaphone value 

I am not sure if this is exactly what is being used (from my observation, it seems to be something more sophisticated), but this seems to do the trick for me. I got a bootstrap template, wrote a couple of controller methods (limonade kicks ass here), and that's it. What you see is the outcome of that work in ~6 hours of hacking. 

Known Issues

1. System is relatively new, so you might not get poems at many chances
2. Sometimes the lines might be repeated in the poem
3. Occasionally search breaks for no reason, little refresh or a new keyword should do the trick
4. No Share features like in 140verses (intentionally not implemented)

Disclaimer - Tweetom idea was inspired from 140verses and the original developers are to be highly appreciated for the innovative thinking + work. Tweetoem was just a self ego satisfying bad hack to replicate the same in less than a day. It is by no means tries to compete with 140verses / their scope. 

Twitter Streaming Limit Workaround

I was working on my final year project (BlueIgnis) which uses Streaming Twitter API. I had the following understandings from them (on free version):

1. One Account can open only One Streaming Connection at any given time
2. One IP may be associated with only One Account while streaming. Rotation of Streaming connections based on multiple accounts are not allowed. May lead to IP Ban. (All the more reason to use EC2 Instances for Streaming :P)
3. One Streaming connection may allow upto 400 tracks (different keywords) to filter from.
4. Reply to 402 Error codes with proper HTTP Status.
5. Should use non-aggressive re-connect policies, must give substantial amount of time in-between subsequent requests.
6. Periodically we must stop the Streaming Connection, add more tracks (keywords) to the list and re-start the connection, rather than individual connections for multiple times.

Based on these understandings, I came up with own Architecture for Twitter Streaming. Below diagram represents the overall architecture of my application with respect to Twitter Streaming Component.

Hosting the Twitter Streaming on an EC2 Instance, we can achieve 400 tracks (keywords) per node which can handle approx. 30 - 50 customers based on my use-case. I periodically (~10 min) check if there are any new tracks that needs to be added to the node until it becomes 400. Since I need to know which user requested the track, which is not possible to get from the current way the Streaming API works. 

So I decided to build a Local Firehouse, where in I stack all the tweets for all the tracks, all in a single location. Then, I use a FullText Search feature of MySQL (my datastore) to search for the related tweets continously so that I can achieve the feel of a bit delayed streaming yet close-to-realtime processing.

If you have any better ways to get things done, please let me know.