Indexing server crash

While I was out catching up with a few people in NYC last night, our main indexing server for decided to completely die…after spending a few hours trying to revive it, I finally gave in and pronounced it.

Of course, it’s the one server that had some code on that was not in version control, or apparently backed up anywhere…so now I’m left with rebuilding the server from scratch (done as EC2 makes this pretty simple) and working from the last version of code I did have backed up (and trying to re-apply all the updates and changes I did since that backup).

The good news is that it’s forced me to think about stuff I’ve had running for almost a year now…and I’ve identified some areas where I can make some serious upgrades and updates to improve the system as whole…the bad news is that while I implement these new ideas and put in all the fixes, the service is down.

These are the just the type of issues you have when you are bootstrapping a project, trying to keep costs as low as humanly possible, and operating with just one or two founders who are over-committed to doing everything to start.

So the pressure is on to get everything fixed up (I expect to have us back and better than ever within the next day)…but this is also the advantage to being a beta project and to not (yet) being a funded company or charging people for anything…being down is not the end of the world and (most) people understand and accept the situation.

Believe it or not, this is the fun part of a startup and these are the moments and the challenges I actually enjoy.

This post has received 47 loves.


This is the personal blog of Kevin Marshall (a.k.a Falicon) where he often digs into side projects he's working on for and other random thoughts he's got on his mind.

Kevin has a day job as CTO of Veritonic and is spending nights & weekends hacking on Share Game Tape. You can also check out some of his open source code on GitHub or connect with him on Twitter @falicon or via email at kevin at

If you have comments, thoughts, or want to respond to something you see here I would encourage you to respond via a post on your own blog (and then let me know about the link via one of the routes mentioned above).