There’s a reason that google became a house hold name, they are amazing at a pretty hard problem - search.
As simple as it sounds to write a good search system, it’s just not. Not if you want it to scale, and not if you want it to be full featured with fast response times.
Take the reviews.com search…
On the backend:
On a predefined schedule Perl hits Oracle to build text files, which are passed off to a verity indexing program (all running on relatively closed Unix systems).
On the front end:
Coldfusion (on a Windows box) accepts a query string from an html post, parses the string to build the proper Verity query syntax, then sends an http request with the newly built query string to a verity web service (hostd on a Unix box), which in turn responds with an XML document back to Coldfusion. Coldfusion then parses the XML document and generates the proper HTML page that is displayed to the user.
That’s a lot going on (with a lot of different languages, systems, and processes), for a simple little search of a fairly small data set…oh and before you ask, no it’s not all that willing to scale, full featured, or fast – but it’s got more features than a pure SQL implementation could easily accommodate (and to be fair this is the same search process that reviews has been running for almost 8 years now!).
Anyway I bring this up, because tonight I’m doing a little updating/debugging to this process and find the biggest challenge simply in tracking down the right spot to focus on. As Indiana Jones once said…X never marks the spot (except when it actually does).
Of course this jumping between various languages and systems is one of the things that helps keep me interested in this career (I’m jumping through many of the same type technology hoops at Bowker during the day light hours)…but there are times when I have to step back, take a breath, and wonder just how I ever get any of this stuff done :)
This post has received 39 loves.
Kevin has a day job as CTO of Veritonic and is spending nights & weekends hacking on Share Game Tape. You can also check out some of his open source code on GitHub or connect with him on Twitter @falicon or via email at kevin at falicon.com.
If you have comments, thoughts, or want to respond to something you see here I would encourage you to respond via a post on your own blog (and then let me know about the link via one of the routes mentioned above).