First a disclosure: I believe there are a few people already working on this idea (actually I know for a fact that at least one company already has a working prototype in this realm ‘in stealth’). Still, I don’t believe anyone is purely focused on it as a stand-alone service or with the specific details I would like to see, so I thought I would dump out my random thoughts for the idea anyway.
First a few important things I believe to be true:
1. Tons of interesting conversations, often even more interesting than the original content that sparked the conversation, are happening within Disqus across the internet.
2. Most search engines (especially Google and Bing) are not currently indexing Disqus comments in a 'useful or meaningful’ way.
3. There is a clear, and successful, revenue model to implement around quality search.
So, what’s the actual idea?
Simple to say, slightly hard to do. Build a search engine that indexes, in a quality way, all the content flowing through Disqus in as near-realtime as possible.
You’ll notice the subtle, but key, thing in the idea is the 'in a quality way’…what does that mean? That’s the tricky part I think.
To me, it would mean do some auto-tagging and contextual grouping (ie. analysis) between the content within a comment, the content the comment was associated with, the content within the thread as a whole, and then of course the relationship/meta-data associated with the commenters themselves.
That is, you need to develop a 'conversation rank’ algorithm.
As you can imagine, this quickly becomes a massive data set problem (with many interesting variables)…so it’s really not a trivial thing to get correct (and there are quite possibly many 'quality’ ways to approach/solve this).
And this is why I also believe that, for it to work well, it probably has to be it’s own standalone service (and not a feature worked into another, larger, service – though a service with a large enough scale might be a 'good enough’ solution for now since nothing else exists).
Doesn’t Disqus already do this? If not, shouldn’t they be the ones to implement it?
Disqus does actually have a search feature built in…but it’s not at all like what I’m talking about here.
(As an aside I suspect that no one really uses the version of search they currently offer up – in fact I urge you to go play with it and you’ll see why).
So while I agree that they *should* be the ones to do it, I don’t think Disqus will actually build something like what I’m talking about/envisioning any time soon.
Beyond that primary focus, they care deeply about the experience for the commenter but from the view of making it easier/better to engage with the blogger (and others around that bloggers’ content).
So I don’t believe they are currently that focused on the data set across the Disqus universe, and the amazing stuff that *could* be done with it.
This leaves a huge opportunity for the rest of us if we so choose to act.
Side note: I do suspect that a successul implementation of what I’m talking about here, even at a small scale, would be a high acquisition target for Disqus. If nothing else, it would force them into thinking about, and acting on the bigger picture, about what people really want from the Disqus data set (ala the Summize deal for Twitter back in the day).
The other advantage that I see around this being a standalone service is that you could include many other data sets around conversations that are not currently being indexed (in any quality way).
So anyway - if I were to be thinking about building a new search engine, this is the angle I personally would be the most interested in working right now (and the one that I think is currently being the most underserved)…
What do you think?
This post has received 37 loves.
Kevin has a day job as CTO of Veritonic and is spending nights & weekends hacking on Share Game Tape. You can also check out some of his open source code on GitHub or connect with him on Twitter @falicon or via email at kevin at falicon.com.
If you have comments, thoughts, or want to respond to something you see here I would encourage you to respond via a post on your own blog (and then let me know about the link via one of the routes mentioned above).