Email to Twitter...

I’m thinking about building a simple little email-to-twitter service…somewhat inspired by this request, but also just because it’s a feature/system I’ve wanted for awhile for myself as well. In thinking about the design a little bit, it seems most likely that I’ll have to build a simple web front end that users will need to initially visit to sign up for the service (I’ll use Twitter’s simple oAuth stuff for each user to approve the app - much like I do with Gawk.it if you want to use the web to log your predictions). Once a user has approved the service, I’ll store the oAuth details associate to that user’s email address…and then I can set up some simple cron jobs to run some monitoring scripts. The first script will need to monitor each user that is signed up for direct messages…when a new one is found, it will simply need to email the user with the details of the direct message. In a lot of ways, I’ll be doing the same sort of thing that I do with fubnub.com because I’ll have to make sure that I only alert users on the most recent DMs (ie. avoid duplicate alerts) and I’ll have try and make sure that the system can scale for a large set of users (just in case the rare event occurs that lots of people actually want to use the service!). So, while I haven’t started it yet, this script should be pretty straight forward for me to hack together. The second script will need to monitor the incoming mail box for messagse that users send and then post the content of that email to twitter for the given user (ie. so people can send an email and have it directly post to their tweet stream). There are actually a couple of ‘gotchas’ to watch out for with this script though… 1. The email address that someone sends a email from will need to match to that of the email address associated to a twitter account that’s granted our system approval…this isn’t much of a technical issue, it’s just something that needs to be identified and communicated well to users up front so they don’t get confused if/when their messages aren’t getting posted to Twitter. 2. Email contains both a subject and a body…what part becomes the tweet? On one hand I can see a case for just using the subject of an email as the tweet (people are more likely to find it natural to keep a subject short and to the point)…but my arguement is that, especially in responding to emails, subjects are often overlooked as the conversation moves on…so I’m most likely going to decide to ignore the subjects of any emails passing through the system and focus just on the body of the emails for the text of tweets. 3. Speaking of the text of tweets, how do you determine what a user really wants to do? The post I mentioned above asks for the service related to the specific idea of responding to direct messages via a simple email reply…but there’s no real reason a system like this, once built, couldn’t be used to post to twitter via email at any time and for any situation (heck I like my iPhone mail set up for composing a lot better than my iPhone twitter app…if I built this service, I would probably defer to emailing my mobile tweets over using the other apps or even texting them to the short code). But this raises the question of just how much to automatically do for the user…if the system was built purely to respond to direct messages…you could determine who sent the direct message for the user and automatically D that user back on behalf of the user…but in a more general system, you will most likely just post whatever the user sends in the body of the system (so to direct the user back, you would need to include the command and user in your email body like “D falicon you totally rock!”). In the end, I think that’s both the easier thing for me to code (yeah) and the better solution for the user (double yeah)…because it’s more true to the Twitter experience, and allows for more flexibility. 4. Twitter has a very strict 140 character limit…but email has no such size restriction. What to do if/when a user sends an email that contains more than 140 characters in the body? This is a bit of a tricky situation, because you know a number of people are going to ignore that limit (either intentionally or just because they don’t know or remember it)…so what my approach will most likely be is to do something like what http://xltweet.com does in letting you tweet more than 140 characters…that is, in the event that the text is going to be over 140 characters, I’ll have to store a copy within my service, create a unique URL to it, and then chop the tweet down to the right size (making sure to include room for the newly generated URL). Of course to do that URL bit, I’ll want to use a URL shortening system (like bit.ly)…and while bit.ly (and others) offer up some pretty solid API, I don’t think I want to clutter up their system with (potentially) millions of references to long tweets…it seems like it would just be much simpler to go ahead and generate my own short URLs that are custom to this project. Taking it a step further, I would basically design a table that stores the long text in a CLOB field (and probably some other data related to the tweet like the tweet id, the sender, maybe the date and such – we’ll see what makes sense if I get around to actually building this out)…this will give me a unique id for each record in the table…and so then what I will need is something to convert that unique id into a query string that’s sure to stay fairly small (so again in the event that millions of these records get generated, the link that points back to the long text doesn’t take up too much of my precious 140 characters). With all of that in mind, and just becaus I found it especially interesting, I went ahead and hacked together this quick Perl program that will take a base 10 integer and generate a unique key that should be a manageable size for quite some time.

# simple key generation script using basic # base conversion (10 to 36) and control Hash (A-Z0-9) use strict; use warnings; sub base_conv {   # convert from base 10 to base 36   my $num = shift;   # the remainder is the value we’ll want to use to look up   # the related character in our control hash later   my $remainder = $num % 36;   # the value is the new number we will use to determine if   # we’ve exhausted our conversion yet or not   my $value = int($num / 36);   return ($value, $remainder); } sub generate_key {   # given a base 10 digit, create a short url key   my $id = shift;   my $continue = 1;   my $slot = 0;   my $key = “”;   my @bits = ();   my %control = ();   my $remainder = 0;   # start by recursively calling the base conversion routine   # building an array of look up numbers   while ($continue) {     ($id, $remainder) = base_conv($id);     # keep track of the remained each time through     # (this will be our lookup keys)     push @bits, $remainder;     if ($id == 0) { $continue = 0; }   }   # build our base 36 control hash (letters A-Z and numbers 0-9)   # start by building with the letters   foreach my $value ('A’..'Z’) {     $control{$slot} = “$value”;     $slot++;   }   # don’t forget the numbers   foreach my $value (0..9) {     $control{$slot} = “$value”;     $slot++;   }   # build the key from the values in our array as keys to our base 36 control hash   # techically, proper base conversion would reverse the array (and then join them)   # but we don’t need that here as we are just using the conversion of each bit as   # keys to a look up hash so the final key in this approach would be the same   # (in reverse order of this). We save the processing power because it adds no   # value to our end gaol.   foreach my $rev (@bits) {     $key .= $control{$rev};   }   return $key } #### Testing Generating a key, and printing the new key #### my $id = $ARGV[0] || 1; my $key = generate_key($id); print $key . “\n”;
I think it’s actually pretty solid, and seems to work in my quick tests…and so if/when I get around to actually building out the service mentioned above, I’ll probably use some version of this approach/script… But let me know what you think…would you approach this in a different way? What about the other points I mention above, am I missing anything or overthinking some of the problems?

This post has received 38 loves.


ARCHIVE OF POSTS



This is the personal blog of Kevin Marshall (a.k.a Falicon) where he often digs into side projects he's working on for digdownlabs.com and other random thoughts he's got on his mind.

Kevin has a day job as CTO of Veritonic and is spending nights & weekends hacking on Share Game Tape. You can also check out some of his open source code on GitHub or connect with him on Twitter @falicon or via email at kevin at falicon.com.

If you have comments, thoughts, or want to respond to something you see here I would encourage you to respond via a post on your own blog (and then let me know about the link via one of the routes mentioned above).