OK I gave in to the massive outcry (and when I say massive, I really just mean the three or four people that pinged me with quick questions about the closing of conversationlist)…Conversationlist is back.
As I mentioned before, the primary reason I shut it down in the first place was that it’s a complete side project…and it had been costing me about $70/month in server fees.
The cost of the server was basically that high because I hadn’t taken any time to optimize the code or the list building process…instead, as the user base grew, I was just upgrading the size of the server I hosted it on (the lazy mans way to scale).
That approach worked fine for quite awhile because it meant that I didn’t have to put anything but a little more money into the project month after month…for about two years.
That is until one day when I sat down to actually review my server costs for my various projects…and I realized that I was spending $70 a month for a dedicated server for conversationlist (a simple, for-fun, side project), and only a few times more than that for ALL the servers that power knowabout.it (an attempt at a real, world-changing-potential, company).
It was clear I needed to either take the time to retool the service, or just kill it…and well, killing it is much quicker and easier than retooling a bunch of old code right?
So I just made a knee-jerk reaction and killed it without much thought.
But the truth is, I’ve always liked the small success that conversationlist has found…and I do think it’s a neat little project with some interesting, yet-to-be-culled, potential.
So this past weekend, when I wasn’t involved in Pack 59’s Pinewood Derby or doing something on that other awesome project I haven’t told you about yet, I spent much of my free time rebuilding the Conversationlist service from scratch.
The orig. version of conversationlist was a simple PHP project that used the Twitter REST API, PostgreSQL, and cronjobs to do everything.
It was really just one big synchronous task that would go account by account, pulling the last 200 tweets the user generated, doing some basic regex code to determine mentions, and then more Twitter REST API calls to get user information…eventually it would remove the members off a list that needed to drop off and then add the ones that needed to be put on.
This worked well enough for the first five-to-six thousand users, but somewhere around that point it really started to fall apart…so a year or so ago (I actually forget exactly when), I did take the time to implement a few minor updates to help scale a bit better.
At that time, I replaced the PostgreSQL datastore with a Mongo instance (in case you didn’t know, I’ve been a HUGE fan of Mongo for the past year or two)…and I updated the cronjobs so that instead of direct PHP scripts, I had Perl scripts do some of the heavier lifting and a bit of threading so that I could process more lists in the same (or less) amount of time.
But the entire web experience was still pure PHP…and ultimately each list was still being built by a large number of Twitter REST API calls (the real bottleneck in much of the process).
Again, this worked well enough…and as long as I had a powerful enough server, it could even scale without too much trouble (but as I’ve mentioned, had lots of potential to get expensive and glitchy too).
So it was finally time for a complete re-thinking of my approach…
Since the time that Whitney came up with the idea for conversationlist (you do know he’s the real brains behind this whole thing right?), Twitter has made massive updates to the things a developer can do…and I also like to think that I’ve made massive strides in how I approach and prepare for scaling successful services as well.
So this time around, the web site is pure Python (using tornado, tweepy, and pymongo)…and, like I do with knowabout.it, I’m taking advantage of Twitter site streams (I customized a version of Perl’s AnyEvent library that I use for all my Twitter Site Streams stuff – just reused that for this update).
Though I will see nice improvements from switching from PHP (behind an Apache2 server) to Python (behind an Nginx server), Site Streams is the real key to the whole thing.
With Site Streams, I can get just the mentions (as they come in, in real time) and each includes all the details I need for adding them to lists (ie. the user’s twitter id, not just their screen name)…so I can keep a running record, in real-time, of what a given conversationlist needs to look like without any extra or time-consuming REST calls.
The end result also means that I can now host the entire service on a reserved M1.Small EC2 instance (that I already was paying for)…and while it’s certainly not fast by any means on that tiny hardware, it is back alive, with the ability for me to introduce some cool new features (if I find time to implement them)…and all for around $20/month right now.
Not a bad price for rebuilding about 10,000 Twitter lists a day (in case you’re wondering why that number doesn’t match numbers I’ve reported in the past…in the process of the re-tooling, I trimmed out the spam and dead accounts which dropped the overall user count from about 14,500 to just over 10,000 active…but we’ll see where that number goes over the next couple of months).
If you take the time to jump over to conversationlist.com, hopefully you’ll notice I also took the time to redesign the web site (using Twitter Bootstrap of course!).
If/when I get the chance to implement a few of the little ‘extras’ I now have in mind thanks to all this work (and now that I actually spent more than 5 minutes thinking about the project again)…I’ll see if I can’t generate enough income from it all to at least warrant an upgrade to an M1.Large instance (the service already could benefit greatly from the server upgrade, but I’m still trying to save on cost and already had this m1.small reserved instance laying around).
So anyway…it’s back…hopefully better…and getting ready to go to the next level…so stay tuned!
This post has received 39 loves.
Kevin has a day job as CTO of Veritonic and is spending nights & weekends hacking on Share Game Tape. You can also check out some of his open source code on GitHub or connect with him on Twitter @falicon or via email at kevin at falicon.com.
If you have comments, thoughts, or want to respond to something you see here I would encourage you to respond via a post on your own blog (and then let me know about the link via one of the routes mentioned above).