Search Applications : Search Startups Are Dead, Long Live Search Startups

A couple of months ago I had the pleasure of moderating a panel at TIECon on the Search Industry.  Peter Norvig,  Google’s Director of Research, made one comment in particular that stood out in my mind at the time.  In response to a question about the prospects for the myriad of search start-ups looking for funding Peter basically said, and I am paraphrasing somewhat, that search start-ups, in the vein of Google, Yahoo Ask, etc. are dead.  Not because search isn’t a great place to be or because they can’t create innovative technologies, but because the investment required to build and operate an Internet-scale, high performance crawling, indexing, and query serving farm were now so great that only the largest Internet companies had a chance of competing.

Priced Out of the Market
While the comment might strike some as self-serving, the fact of the matter is that it is true.  Any start-up trying to displace Google, Yahoo, or even MSN or Ask (or for that matter any VC trying to fund them) should just get in their car (or hop on a plane) and go look at Google’s new server farms at The Dales in Oregon.  And if that doesn’t convince them they should head up the Columbia river a bit and check out Microsoft and Yahoo’s digs.   The costs to compete in core search , are now simply to high.

Start-up Salvation: Search as Infrastructure
So where does this leave search start-ups?  For many of them it will undoubtedly leave them in the Web 2.0 Deadpool, but some of them may thrive if they adapt to the new reality. This new reality is that search innovation will increasingly be about applications and not about the core infrastructure.  In fact, there’s a good chance that much of the core infrastructure will be available as a service to search-based applications.  Amazon is pioneering this “search as a service” with its opening up of the Alexa crawling and indexing APIs.  While in the blog world, Technorati already provides APIs for developers to integrate their blog search results directly into applications.   Even Google appears to be taking tentative steps in this direction with Google Co-Op which enables users to limit Google’s Search engine to a customized set of sites and keywords.

One can imagine a world in the not to distant future in which an application designer can easily leverage the billions of dollars being spent by Google, Yahoo et. al., by having programmatic access to what is essentially a custom crawl list and a highly filtered index.  In this way search engines, in some respects, may become an infrastructure layer not too dissimilar from the telecommunications networks and internet standards that they themselves are built upon.

For start-ups, having core search become just another part of the Internet’s infrastructure is actually great news.  This frees them from the huge capital costs required to build a competitive core search platform and instead lets them focus on building a great consumer/enterprise application.  The possibilities here are endless and will undoubtedly result in far more innovations than if search engines remained closed systems.

Seeing the Forest For the Trees
While there will undoubtedly be opportunities for start-ups to extend and improve core search (which is where the vast majority of effort has been extended to date), some of the most interesting opportunities will come not from trying to improve the accuracy and context of a single query, but from looking at aggregate information about search indexes, results, and queries across time.  In other words, the marriage of this search infrastructure, with persistent queries and advanced analytics will likely create an entirely new class of applications that generate insights and create value not by finding the specific piece of information someone is looking for, but by analyzing the ebb and flow of information across the web.   It is here, in this new world of search applications that more than a few start-ups are likely to find a happy home.

In my next post I will talk about some of the more promising search applications that may develop in the next few years.