A little sidebar on Conversationlist

Conversationlist has been up and running for a little over six months now…while our initial release saw massive adoption thanks to some attention from Fred Wilson and Ev Williams things eventually calmed down a bit…and now we only see a few new users a day (well for a few days/weeks we saw no new users but that was because I moved the system to a new server to handle some scaling issues and it turns out I blew some important pieces up without noticing until just today!)

Still, with the ups and downs (all thanks to my own laziness or half-brained-ness…or a combination of the two) we are currently at 10,313 accounts (of which 6,832 are active lists) and we’ve logged over details on 450,875 twitter users with over 220,099 unique conversation connections (this number is actually a little lower than it once was because I wiped it during the server move and started over)…

When you look at the Twitter universe as a whole, this is a pretty small subset…but I still think it’s a pretty interesting one, and I think there are a lot of interesting things that can be done with this information (though I haven’t had any time to actually sit down and do any sort of analysis on any of it).

I’m not sure if there’s much potential in monetizing any of this directly…but I didn’t really build conversationlist.com with any idea/plans of monetizing it (I just built it because it was an idea Whitney had, and I could do it in about a day…so why not?)…but it does seem like, with a little effort, there’s gotta be some interesting things that *could* be done to make a little cash with it (who knows, maybe some day I’ll sit down and figure that out).

Anyway, from a tech. point of view, the most interesting thing about the entire project is really the concept of the daily rebuild…from the start I designed the entire system to really be a web service…so whether you are building your list via the web site, or our it’s getting rebuilt via our nightly batch, it’s really just calling a web service that does the work of figuring out if you’ve got a list, if it should be rebuilt, and who should go on that list…and, partially because of twitter rate limiting, it was designed from the start to use your own tokens and keys to rebuild your list (so if you remove conversationlist access via your twitter settings, we can no longer rebuild your list for you).

So the initial nightly rebuild was really just a perl script running on the server once a night…all it did was query for the list of accounts that were active, and then sequentially go through and call the web service for each…and this process worked great up until the daily build got to be about 3,000 lists…

Once we hit that tipping point, the sequential process still worked, but mostly thanks to the speed of the database (it’s a PostgreSQL database behind the scenes), it was taking close to 24 hours (sometimes over) to get all the way through the building…which meant it wasn’t really doing it every day for everyone…

So I had to update it to be a bit more distributed…and since really all it does is call a web service, this was a pretty simple thing to do…so now instead of one Perl script that does everything sequentially, I now do the following process:

1. Once a day, a perl script creates a collection of text files…each text file is just a list of 200 accounts that need lists rebuilt.

2. Every 15 minutes, a cron job kicks off another Perl script…that script grabs one file from the directory (if there are any there) and then starts building the lists for the accounts in that file (and deletes the file when it’s done).

Though I could probably speed this up even more by threading the script in step #2…it hasn’t been needed yet (though it will eventually get there I’m sure and then I’ll do it)…and honestly, while I can, I like limiting the amount of server traffic/processing that the rebuilding causes.

So anyway, though I don’t talk about it much…conversationlist.com is still up, still chugging along, and still presenting some interesting challenges here and there…and these days, that’s what I call a successful project!

This post has received 43 loves.


ARCHIVE OF POSTS



This is the personal blog of Kevin Marshall (a.k.a Falicon) where he often digs into side projects he's working on for digdownlabs.com and other random thoughts he's got on his mind.

Kevin has a day job as CTO of Veritonic and is spending nights & weekends hacking on Share Game Tape. You can also check out some of his open source code on GitHub or connect with him on Twitter @falicon or via email at kevin at falicon.com.

If you have comments, thoughts, or want to respond to something you see here I would encourage you to respond via a post on your own blog (and then let me know about the link via one of the routes mentioned above).