Burnham's Beat

Infospace and the Great Shareholder Robbery of 2007

Wow.  Infospace reported earnings today and the stock was off 14%.  But that's not what's making me say "wow".  What's making me say "wow" is that I took this opportunity to take a look at Infospace's 2007 "earnings" report and I have to say I am impressed because it has to go down as one of the great shareholder robberies of all time.  Infospace, as you may recall, was a once high flying internet content company that assembled a motley menagerie of web and mobile based content businesses via 30+ acquisitions over the last 10 years.  They bought everything from Authorize.Net, to Go2Net, to Switchboard.  While there was supposedly a grand strategy driving to all these purchases, arguably what they were left with after 10 years and $1.7BN in paid in capital was just over $1BN in retained losses and a motley collection of business that looked like they were going no where fast.

Anyway, the company's mobile business was held out as the potential savior for a long time but as that too began to implode in 2006 and early 2007 the management team basically threw in the towel and embarked on a process of selling off the company's assets piece meal to the highest bidder.  No doubt a somber occasion given that the sales represented the culmination of a failed strategy that had cost shareholder's a cool $1BN, but at least you have to give the management team and the board credit for doing the honorable thing by admitting they failed and doing their best to salvage something for the shareholders.  Or do you...

On Closer Inspection
Before you give the management team and board any credit for doing the right thing, you might want to know a little something about the a deal they struck for themselves that nicely coincided with their fire sale.  The plan was to sell off the assets and dividend out the cash proceeds to the company's long suffering shareholders, which sounds fair enough.  However, the management team was able to convince the board that they should get paid a "bonus" equivalent to the dollar value of the dividends that would theoretically accrue to any vested or unvested stock options they might have.   So if the company did a $5 dividend of sale proceeds and the stock dropped $5 (which it inevitably would), the management team would get a $5/share bonus for each vested and unvested stock option they owned.

Now on one level that sounds fair enough.  I mean after all, why should the management team and the employees have their options go further underwater simply because they are doing the right thing and trying to get shareholders back some cash.  However the net effect of such a scheme is to basically give the management team a gross cut of whatever they sell an asset for without regard to whether or not the sale was even profitable.   The cynic/economist/anyone with common sense would say that under such a incentive structure management would race out and sell everything and the kitchen sink for whatever price they could get because it was money in their pocket no matter what.

Care to guess what happened?

Everything Must Go!
That's right, Infospace's management team ran out and sold everything they could for whatever price they could get.  The directory business, painstakingly built up over a period of 10 years: sold for $225M to a private equity firm.  The mobile business, which they been acquiring new businesses for less than a year earlier: sold for $135M to a private competitor (who reportedly now mightily regrets the purchase).

And what did they do with all that cash (as well some cash from settling a lawsuit with a former founder who defrauded the company's investors)? Why surprise, surprise, they dividended out all that cash out to their shareholders in two special dividends totaling $15.30/share or over $500M in cold hard cash.

And what, pray tell, what did the management team get for the arduous task of lifting up the phone and calling their bankers?  A cool $90M in special cash bonuses and stock compensation in 2007.  If you are on that management team, the thought that likely came to mind as you cashed your 2007 bonus check was "God Bless America!"

Defending the Indefensible
But wait, defenders of the management team's "golden dividend" might point out that the asset sales generated a combined $149M "gain" and surely the team should at least be entitled to share some of that gain.  But that logic fails to account for that fact that prior to 2007, Infospace had already taken $240M in impairment charges since 2000 which means that the actual "gain" on sale of those assets was likely far lower and might even have been a net loss of over $90M on an original cost basis. (I'd guarantee its a loss, but it's impossible to figure out what impaired assets were sold given how many deals they did.)

One final point of defense might be the stock price.  Defenders might point out that at the beginning of 2007, before management embarked on the asset sale strategy, Infospace's stock price was $20.51, and even though it closed at $10.38 today, when you add back the $15.30 in dividends, you get an adjusted price of $25.68 or about $5.17 higher, so one could argue that despite all the management payouts they still created value.  There are only two problems with that logic:  1.  With 33 million shares outstanding, $5.17/share translates into only $170M in increased "shareholder value" vs. $90M in management comp which equates to a 35% cut for the management team.  M&A bankers would sell their mothers to get a 3% cut of a sale so I shudder think what they'd do for a 35% cut.  2.  If you go back just 3 years, the stock price was at a whopping $47.55 meaning that shareholders have suffered a 46% loss in the last 3 years, but the management team and the board thinks that's an occasion for them to give themselves a windfall payday the size of which would make even an investment banker blush.

Don't Look Now But You've Been Robbed
Let me be clear: I've never owned or shorted (unfortunately) Infospace stock, so I don't really have a dog in this fight.  I am just flabbergasted that none of their shareholders stood up to such a blatant scheme and called it for what it is: highway robbery.  I don't have a problem at all with paying management teams well for creating value,  I just have a huge problem with handing out huge bonuses (i.e. more than 10X the already egregious fees charged by bankers) to people who sell assets for little if any gain when their shareholders are already sitting on $1BN an losses and a 46% stock depreciation since 2005.  Call me crazy, but I have a problem with that even if, inexplicably, Infospace's shareholders don't.  You may also ask why didn't the board, who supposedly represents the shareholders, have problem with that: simple, they cut the same deal for themselves.

As for shareholders, they are now left with a business whose only significant asset is a website called Dog Pile.  If you're like me, the image that pops into your mind when you hear the term "Dog Pile" isn't exactly a pile of money, which at least strikes me as poetic justice if nothing else.

I currently have no investment position, long or short, in Infospace.  This is not investment advice, it is an incredulous rant.  Please see my disclaimer at the bottom of this page for further details.

Wigix: An Idea Whose Time Has Finally Come

Wigix is launching its public beta today and the world of online auctions, indeed the world of "stuff" in general may never be the same.  As an early investor in Wigix and an avid fan of its team and concept I want to congratulate the Wigix team on all the hard work that has gotten them to this point!

What exactly is Wigix?  Wigix is, at its heart, an ambitious attempt to bring online auctions into the 21st century by applying modern market technology to everyday items.  In essence, Wigix creates a stock market for your stuff.  Into this stock market Wigix then weaves a deep sense of community and persistence which then enables people to track, discuss, and share information about their stuff. 

The ultimate, rather immodest, goal of Wigix is to create a complete record of every item in existence, who owns it and what it's worth, as well as a platform that enables people to track, discuss, and trade all of those items.

Ever wonder how much your stuff was currently worth?  Or what other people paid for their original iPod?  Or what owners of the PS3 think is its worst feature?  Or what interesting things your friends just purchased and why?  Wigix is a platform that can provide answers to all these questions as well as an online auction experience that is light years ahead of what's available on the Internet today.

Despite just entering its beta launch, Wigix is already an incredibly rich site with a huge amount to see and do.  It’s still a beta though, so there a few features left to add and a lot of content still to come.  I could go on and on about Wigix and why I believe it’s one of the most interesting web platforms that has launched in a long time, but you’d be better served by just jumping over to the site and taking a look around for yourself.

Top 10 Things to See and Do On Wigix
Before you go though, let me give you my list of the Top 10 things to see and do on Wigix as a way to highlight some of the most interesting aspects of the site:

1. SKUs: Wigix does not have listings, but rather SKUs.  SKUs are kind of like a pre-built listing or a stock ticker for a specific item.  Users simply indicate which SKUs they own by placing these SKUs into their portfolio.  Once in their portfolio, a user can track the value of that SKU and then decide to sell it at any time in the future.  Once you see a SKU and compare it to a listing at a traditional online auction site, you will never want to go back.  For example, check out the SKU for the iPhone and look at the all the rich detail and information that is available.
2. Catalog Browsing:  At the heart of Wigix is a catalog of SKUs, roughly 462,000 and growing.  These SKUs are necessary to enable trading, but they also create what is in effect a huge catalog of stuff.  Try browsing through the catalog. It is a very cool and visually appealing experience thanks to some nifty programming by the team.  I think Wigix’s browsing and search features have set a new standard for product catalogs on the web.
3. Persistent Portfolios: Wigix does not charge listing fees but rather encourages all users, not just buyers and sellers, to build and maintain portfolios of their stuff, similar to stock portfolios. These "stuff portfolios" persist over time allowing users to not only track market prices for their stuff but to track the stuff accumulated by friends, family, and fellow collectors.  Try it out by registering and adding a few things you either own or want to own to your own portfolio and you’ll immediately how easy it is.  Here is my public portfolio of stuff (you need to register to view it), feel free to make a bid if you see something you like!
4. “Ask the Owners”:  Ever have a very specific question about a potential purchase but have been unable to find the answer on the product’s website or in product reviews?  Because Wigix keeps track of who owns what, it’s possible for any user to ask the owners of a particular product a specific question.  If they want to, the owners can then respond with an answer.  I did this when I wanted to know what Wii game I should I buy for my nephew.  Check out the responses on the Wii SKU page under the “Ask the Owners” tab.  Very cool.
5. Bid/Ask Trading: While the NASDAQ uses a bid/ask trade system to trade stocks, Wigix uses a bid/ask trade system to trade stuff.  In comparison, EBay uses what are called "English Auctions".  Wigix's bid/ask system enables instant trading, continuous historical data, and is not subject to "shilling" or "sniping", two problems that are endemic to EBay.   It’s interesting to note, that many stocks and bonds were also once traded using English Auctions, but once bid/ask-based exchanges opened, the purveyors of these English Auctions quickly disappeared.
6. Community : Wigix has woven community and social networking technologies directly into the main features of the site.  This makes it possible for people to keep track of what friends, family, and fellow collectors own or want to own.  It also makes it possible to interact with the owners of a particular item and join groups of like minded individuals.  If you register, make sure to add me as a friend (my user name is “Bill”) to get a sense of the community features.
7. SKU Owners:  In the same spirit as Wikipedia, Wigix’s catalog of SKUs can be expanded and enhanced by any user.  Wigix rewards SKU contributors with a small share of the revenues generated by that SKU.  Try adding your car or cell phone (if they aren’t already in the catalog).
8. Category Experts:  Manging all of the SKUs in a particular category is the responsibility of category experts.  Who are the category experts?  Pretty much anyone who has the passion and inclination to apply.  Like a guide at About.com, Category experts get a small revenue share of all the revenues generated by the SKUs in their category.  From a technical perspective, what’s cool about both SKU ownership and the category experts is that to enable those features Wigix has had to create to what amounts to a massively distributed database that can support thousands of what are, in effect, DBAs.  I am the current Category Expert for Audis, Digital Media Players and NAS Drives.  If you are really passionate about a particular category of stuff, apply to be an expert.  It’s not much work and it’s a lot of fun being master of your own little corner of the database.
9. SKU Trading: Once you add a SKU to the catalog and your submission is approved by the appropriate category expert, you own the SKU and not only are entitled to a share of the revenue it produces, but you can actually trade ownership of the SKU, in the same way that you can buy and sell a domain name.  Now the value of a SKU is very speculative at this point because they won’t have any value until they start generating revenues, but in the meantime it is fun to speculate and trade them.  In fact I picked up the Blackberry 8700c for $1 this morning although someone just bid $30 for the iPhone SKU (I had bid $1), so it looks like I won’t be getting that one…
10. Facebook Apps:  Yes like any good start-up these days, Wigix has its own Facebook apps.  My favorite is the MyWigix application which allows me to display my portfolio of stuff on my Facebook profile and automatically creates new Facebook feed items when I get new stuff.

Anyway, as I said before, I could go on and on (and to some extent already have), so to keep this post managable I will end things here. I think I’ll write another post in the future that points out from a business model and technology perspective how Wigix differs from today’s King of online auctions, EBay, but today I just want to congratulate the Wigix team for reaching this milestone.

SkyGrid and the Emergence of Flow-Based Search

GigaOm had a post today on a company called SkyGrid and its official company launch.  As an investor, advisor, and beta-user of the platform, I thought I would chime in with my own self-serving post mostly because I wanted to talk about the advanced technology and architecture behind SkyGrid and why it makes the company such an interesting case study in the evolution of search technology.

Simply put, SkyGrid represents a massive and exciting departure from traditional search architectures and technologies.  If I had to sum it up in a word, I would say that SkyGrid represents what I consider to be one of the first "flow based" search architectures, while traditional search engines are "crawl based" architectures.

Old Search: Crawl/Index/Query
While the technical departure was necessitated by the leading edge demands of investment professionals, it was these needs, and the lack of traditional search's ability to meet them, that exposed some of the most glaring weaknesses of traditional search technology. Specially, traditional search technology and architectures suffer from several glaring weaknesses:

1. Crawl-based: Current search architectures collect information to index primarily by employing massive farms of "crawlers" that systematically crawl IP address spaces. The benefit of crawling is that it is exhaustive, the drawback is that it time consuming and expensive.
2. One-off: Search platforms are designed around rapidly processing one off queries. This makes search engines highly useful and adept at finding "the needle in the haystack" but very cumbersome to use in situations where one just wants to get new results to the same old query.
   
3. Batch-based: Page rank and the other "secret sauce"algorithms behind most search engines today require a very expensive and complicated indexing process to be performed on "snap shots" of data. It can be days or even weeks before newly published content is crawled and properly indexed, meaning that most search engines fail to provide "real time" results for all but the most popular content sources (which they crawl very frequently).
   
4. Unabridged: Search engines are exhaustive in that they return every URL that mentions a string. This is good is you are looking for a needle in the haystack, but bad if you are trying to search on a common term such as "Google" or "Microsoft". While ranking algorithms do a great job of ordering results according to likely relevancy, they don't filter down the number of results. Since most users don't go past the first page of results, this makes it quite easy to miss relevant information that for some reason doesn't rank in the top 10 results.
   
5. Unstructured: Search engines typically present query results as a simple list without context or analytics, beyond say separating them by a simple criteria, such as text and images. While some progress has been made in terms of trying to cluster results or help users filter them, by and large, users still just get an unprocessed, unanalyzed data dump when they do a search.
   
6. Retrospective: Search today is focused on determining what has happened in the past. Who wrote what, who said what, etc. However this does little to help people figure out what will happen in the future.
   

Without giving away the farm, SkyGrid represents an exciting departure from the search technologies and architectures of the past. This change has been made possible by several factors including the widespread deployment and adoption of ping servers and RSS/ATOM feeds, dramatic improvement in several areas of artificial intelligence and unstructured data analytics, and new stream-based methods of database and query design.

SkyGrid Search: Flow/Filter/Analyze
When you put all of these technologies together, along with a laser like focus on solving some of the unique high-end demands of investment professionals, you get a radical new search architecture and technology that not only solves some very pressing and pragmatic problems facing investors, but holds the potential to actually predict the pattern and influence of idea/meme propagation throughout the internet and from there into the financial markets and beyond.  

Specifically, SkyGrid's search architecture differs from traditional search engines in that it is:

1. Flow-based: SkyGrid treats the web as a giant pub-sub system or at least it does to the extent that the rapidly growing RSS/Ping server infrastructure does. It does not crawl the web, but rather the web "flows" to it.
2. Persistent:  SkyGrid persists queries over time so that incremental results are delivered with no additional action by the user. One can easily see how this would be valuable in the case of something like, oh say, a stock, which persists from day to day.
   
3. Real-time: Rather than using batch-based indexing, SkyGrid uses a real-time stream-like query system that queries (and analyzes) new content as it flows into the system. This is particularly useful in situations, such as investing, where a few minutes or seconds, can make a huge economic difference.
   
4. Filtered: Rather than presenting results as a data-dump, SkyGrid uses advanced analytics in the form of entity extraction, meta-data analytics, and rules based AI, to quickly analyze and append additional meta-data to incoming information. This enables users to easily filter data according to number of criteria which greatly lessens the chance of "data overload" and greatly improves the chance of "data discovery".
   
5. Analytical: By applying highly advanced artificial intelligence, such as natural language procession, entity extraction, etc. SkyGrid is able to actually analyze and assess the actual content of a URL, thus enabling it to make determinations such as the sentiment (positive/negative) of information, its "velocity" and its "authority". This goes a step beyond simple meta-data filtering to creating real insights into the content.
   
6. Predictive: SkyGrid's flow based architecture and advanced analytics enable it to view the web as a living breathing, changing entity. By observing the propagation of information over time and across downstream nodes, SkyGrid is in a position to not only assess the "authority" and "influence" of individual nodes, but it should ultimately be able to make reasonable predictions about which information will flow where on the web. By correlating this observed "flow" over time with observed movements in things such as, oh say, stock markets, company sales, etc. it can not only assess the historical sensitivity of changes on the web creating changes in the real world, but it should ultimately be able to theoretically predict, with reasonable accuracy, many of those changes. Yes, I said it: SkyGrid and its new search architecture may ultimately predict the future.

I realize that the last point is at the very least hyperbolic and at worst disingenuous, but as an early beta-user I can tell you first hand that once you see it in action and understand the architecture, predicting the future, in some very specific, limited, yet potentially highly valuable ways, is certainly not something beyond the realm of reason and indeed something that seems quite possible given the progress to date. That said, SkyGrid is still a beta platform and many features have yet to be implemented in part or in full, but the promise and potential is undeniably there.

Google Roadkill?
Why won't SkyGrid simply be put of business by the big players like so many other search oriented start-ups? First and foremost because SkyGrid is delivering a premium product to a group of users that will pay significant sums for something that not only dramatically improves their daily productivity but holds out the promise of providing insightful, market oriented analytics that they simply can't get elsewhere. Second, the existing search engines cannot compete effectively against SkyGrid because to do so would require a reengineering of their basic search architectures to address all of their shortcomings relative to SkyGrid. Moving from a traditional crawl/index/query architecture to a flow/filter/analyze one is a decidedly non-trivial undertaking, one that would require an entire re-architecture of their core services and thus one highly unlikely to be made.  

Well then does that mean that SkyGrid will put the "legacy" search engines out of business? Not at all. The current search engines are optimized to deal incredibly well with the vast majority of queries from the vast majority of users and they will likely continue to do so for some time. Next generation flow-based platforms such as SkyGrid are, by design, tackling a subset of the available queries, but arguably a very valuable subset. Indeed that's why SkyGrid can charge $500/seat/month for its services while the existing search engines must give away their services for fee and make their money on advertising.

Now I can see a lot of people being skeptical after reading this about both my ability to impartially judge SkyGrid's next generation search technology as well as its market potential. To them I would say: just keep your eyes out for some announcements over the next month as I think they will conclusively demonstrate that a number of people far more knowledgeable and accomplished than I see the same potential.

Microsoft/Yahoo: A Bad Deal For Silicon Valley: Take II

Marc Andreessen has posted a very thoughtful rebuttal to my argument (as well as Fred's and few others) that the Microsoft/Yahoo deal is a potentially a bad thing for Silicon Valley.  The funny thing is that I actually hadn't noticed the post yet in my feeds but it was brought to my attention by a number of people who were basically like "Oooo, you've been served!" and were wondering if I was going to challenge Marc to some kind of blog-off or something.

I hate to disappoint folks, but after reading Marc's post I actually agree with most of what he has to say, especially his overarching message to start-ups, which I took to be "Focus on building a great business and the exit will take of itself".  Marc also made a number of other good points around the M&A environment which if I could sum it up were basically "Hey, life will go on, other companies will try to take up the slack, it's not the end of the world."  In particular I think his point that a combined Microsoft/Yahoo may prompt some second tier firms to increase their M&A is a good one.  I also agree that over the long term, creating a big bureaucratic behemoth such as Microsoft/Yahoo is a good thing for start-ups because it means that start-ups will likely be able to dash ahead of the lumbering giant and secure fresh new areas of opportunity well before the folks at Microhoo file even their TPS reports and get out of their staff meetings.

That said, I still think that Microsoft's acquisition of Yahoo is still a net negative from an M&A perspective.  Yes, it's certainly not the end of the world, but on the whole and on the average it's never a positive thing to have an active, well endowed, acquirer removed from the mix.  Yahoo may not have been buying 50 start-ups a year, but they were still one of the most active Internet acquirers not just in terms of deals, but also in terms of bids.  Indeed the most important party in any deal is not the actual buyer but the second place bidder and Yahoo had seemed to make a career out of being the second place bidder lately.   Finally, thanks to its huge market capital, massive traffic and strong (although not relative to Google) monetization platform, Yahoo is one of the few Internet acquirers who have the luxury of being able to easily drop $50-$100M on a "feature" without really thinking about it.  I totally agree with Mark that if you are building a "feature" with the intent of getting acquired by Yahoo or whoever, you were likely doomed to failure a long time ago, but at same time, the cynic in me has seen a lot of "features" get funded in the valley over the past two years often under the assumption that if they get enough eyeballs one of the big three M&A fairies will swoop in and drop $100M just to "keep up with the Joneses".

So I agree that life will go on in the valley and there are some real positive non-M&A aspects of the deal for start-ups, but at the same time, I think net, net it's bad for the M&A environment.  That may change over time as new companies emerge to take up the slack, but over the next 24 months things could be a bit rough because not only will you have Microsoft and Yahoo thoroughly distracted,  but IAC is going to be a complete mess due its dispute with Liberty and AOL appears consumed with consummating its death spiral within Time Warner.  I am sure M&A bankers will do their best keep the deals flowing, but if you have an Internet start-up, given the turmoil within the big acquirers and the rapidly deteriorating economic environment, as Marc suggests, you should definitely just keep you head down on focus on building a real business.

Microsoft/Yahoo: A Bad Deal For Silicon Valley

There's a ton of discussion today about Microsoft's unsolicited bid for Yahoo.  Much of the discussion focuses on whether or not the deal is a good thing for Microsoft, Yahoo or Google's shareholders.  While it's possible it could be a good or bad deal for one, the other, or all three, one thing is for sure:  this a bad deal for Silicon Valley start-ups and their VCs.

How could that be?  Because by swallowing up Yahoo, Microsoft will be removing one of the biggest and most active acquirors of start-ups in Silicon Valley.  The intense competition between Microsoft, Google, and Yahoo has arguably been one of the main factors helping drive up M&A activity and prices for internet related start-ups.   It seems like every rumored acquisition over the past few years has had all three fighting in some way to win the deal.

Even though Yahoo has been wounded of late, it still had a market cap in the 10's of billions of dollars which allowed it to be a legitimate competitor for any deal under $1BN and in fact Yahoo has been a pretty active player in that market whether its del.icio.us, flickr, Rivals, etc.

If it's acquired by Microsoft, that will leave only two Internet media/search acquirors with the ability to easily do sub $1BN deals.  What's more, while Microsoft has recently show a willingness to deal really big deals such as Acquantive and now Yahoo, it has traditionally been less willing to smaller "tuck in" deals, deals that Yahoo has traditionally been much more active in.  Indeed, Microsoft has traditionally been dismissive of these deals because they just don't move the needle for them and their engineering staffs still retain a relatively high degree of NIH attitude.

Losing one of the Valley's most reliable "tuck in" acquirors and second place bidders is a net negative for the Valley.  It will make M&A less competitive in general and will reduce the # of potential exits for "me too" start ups" to 2 instead of three.  That's bad news for Internet content/search start-ups and their VC backers anyway you look at it.

Top 10 Internet Stocks of 2007

2007's list of top performing Internet stocks provides a little bit of  de ja vu from 2006's list.  As was the case in 2006, China remained a hot investment sector, although this year's China winners are different than last year's.  Three of last' year's Top 10 performers repeat in 2007, something to be truly admired in the rough and tumble Internet sector.

To qualify for this list the company had to start 2007 with a market cap of at least $50M and its business had to be focused on the Internet.

So, without further ado, here are the Top 10 Internet Stocks of 2007:

1. China Finance Online
   Price Change: 392% Ticker: JRJC
   Comment:  The Chinese stock market was "en fuego" all year and what better way to play that market than an Internet company that sells online financial services products and information.
2. Baidu.com
   Price Change: 246% Ticker:BIDU
   Comment:  China's #1 search engine appeared to be pulling away from the pack in 2007 despite desperate attempts by Yahoo and Google to get in the game.  Right now the market seems to think Baidu is going to win.
3. Tradus
   Price Change: 184% Ticker: TRAD.L
   Comment:  Formerly known as QXL and a serious competitor to EBAY in Eastern Europe, QXL's stock was supported by takeover rumors all year, which turned out to be true in December when South African media group Naspers announced a deal to acquire it.
4. Acquantive
   Price Change: 168% Ticker: AQNT
   Comment:  Aquantive was acquired by Microsoft in the middle of the year after they scrambled to respond to Google's acquisition of privately held Doubleclick.
5. Priceline
   Price Change: 163%% Ticker: PCLN
   Comment: Last year's #10 performer, Priceline is back again, a very impressive performance for what was thought to be a mature company.  Priceline continued to school the rest of the online travel field, especially with its expansion into Europe.
6. Omniture
   Price Change: 136% Ticker: OMTR
   Comment: Last year's #7 stock moves up one spot to #6.  After a successful IPO in 06, Ominture grew quickly in 07 and announced the acquisition of its main public competitor (VSCN).
7. Amazon.com
   Price Change: 135% Ticker: AMZN
   Comment: The granddaddy of online retailing flexed its growing muscles in 07.  Thanks to growing revenues and somewhat decreased tech spending margins expanded quickly and earnings accelerated.
8. SOHU.com
   Price Change: 127% Ticker: SOHU
   Comment: One of China's largest portals saw impressive growth and benefited for investor appetite for all things China.
9. Gigamedia
   Price Change: 92% Ticker: GIGM
   Comment:  Last year's #2 stock, holds on to a  Top 10 position.  Unlike  other gambling stocks, Gigamedia's focus on Asia as well as its casual games help it buck the trend and keep growing without regulatory interference.
10. BlueNile.com
    Price Change: 84% Ticker: NILE
    Comment: The Internet's largest jewelry retailer saw its stock take off in 2007 and it reached critical mass and began to drive impressive margins.
    

Google's Postini Buy Has Some Interesting Implications

It was announced today that Google has purchased Postini for $625M in cash.  Before I give a few thoughts on the deal I want to congratulate Ryan McIntyre who I used to work with at Mobius Venture Capital.  Ryan made the initial investment in Postini and staunchly supported the company.  He has a good post up on the acquisition that you can read here. 

In terms of the deal itself I think this deal has several implications:

1. Make no bones about it, Google is definitely going after Microsoft's Exchange franchise.  With the addition of Postini, if they just add push e-mail support and an off-line client, GMail will basically be as good, if not better than Exchange.  I haven't heard anything about an off-line client for GMail but as Zimbra has demonstrated, it's possible so I would expect to see something like this from GMail in the near future.  Speaking of Zimbra, IBM should buy Zimbra and kill off Notes.  Zimbra is the only chance it has of keeping up with MSFT and GOOG in the e-mail race at this point.
2. This is not good news for other anti-spam companies, especially if GOOG decides to offer Postini's basic anti-spam services for free.  The way Postini works (you simply redirect you MX traffic to go through their servers) reminds me of how Feedburner works.  Google started offering all of Feedburner's services for free shortly after they acquired them, so it would be logical to suspect that some kind of price cut, potentially all the way to free is coming for Postini's services.  That's not good for other folks out there trying to make a profit on their anti-spam services.
3. If you look at Google's recent enterprise oriented acquisitions they are building a pretty compelling set of hosted applications.  Not only do they have e-mail covered but they have the big three productivity apps (word processor, spreadsheet, presentation) covered as well.  They also added in Wiki capability (via JotSpot).  To link all this together all they need is a hosted file server and integrated search across all these apps (basically a hosted version of Google Desktop).  That's a pretty powerful suite of services.
4. Over the years, Postini received lots of inbound M&A interest, but the only bid they hit was Google's.  I think this speaks to two issues that other companies attempting to compete with Google in the M&A market have: 1) Google not only can afford to pay more for companies, it does pay more.  Google's competitors simply can't compete with Google's prodigious cash flow, multiples, and willingness to take dilution.  2) Google not only pays more, but is perceived as a better place to work by most targets.  Thus, the management team is happy to support a Google deal because they know their team will be happy.   This does not bode well in the short term for Google's competitors.

Contributors Wanted: M&A & IPO Transaction Lists

The other day I published some lists of M&A and IPO transactions in the Software and Internet industries.  A reader made an excellent suggestion that I should turn those lists into Google Spreadsheets and then allow other people to contribute to and use the lists.  This is exactly what I have done and I am now inviting anyone else that would like to contribute updates/edits to the list to e-mail me (just click the "E-Mail Me" link on the left) and I will add you as a contributor to the list.  In return for becoming a contributor you will get several privileges:

1. You will be able to update, edit, and expand the various transaction lists as you see fit.
2. You will be able to use the transaction lists as you see fit.  This includes republishing them on your own blog or website, turning them into an RSS feed, and manipulating them via Google Spreadsheet's APIs. Go crazy, I don't care.
3. Invite other people to contribute and collaborate with the rest of us.

To contribute you will need a Google ID, if you don't have one you can sign up here.

Hopefully we can harness the power of our collective efforts to provide some pretty useful transactions data that will be useful to a wide variety of people.  BTW, I reserve the right to kick out any "contributors" who are free loading and just trying to pick up good content for their site.

As an aside, it's interesting to think of how projects like this may dis-intermediate some of the data brokers that collect, collate and resell this data (such as Factset).  For example, in the financial services industry there are several services that re-sell transactions data to people.  Those folks had better hope projects like this don't take off!

Software and Internet IPO and M&A Lists

Every month I keep a record of all significant public company activity in the sectors I am most interested  in: Software & Internet.   I keep records of all IPOs in those sectors as well as all public company M&A.  I've decided to try out Typepad's relatively new "page" feature by open sourcing these transaction lists.

The links should be good forever and I will try to update the lists each month as I already do this internally.  The lists start in 2004 and are current as of 5/30/07.  Recently announced M&A deals are not listed because they have yet to close.  It's kind of fun to take a walk down memory lane, and see just what has happened over the last few years in each sector.  So without further ado here are the lists:

Internet IPOs
Internet M&A
Software IPOs
Software M&A

If you see anything missing or anything that needs a correction just e-mail me.

Yahoo Buys Rivals From $75M More Than Their 2001 Offer

Yahoo apparently agreed to buy Rivals.com for $100M today and that brought me back down memory lane a bit, so I thought I would share.  In Q1 2001, Rivals was hemorrhaging cash and appeared to be on its last legs.  The site was still attracting a lot of traffic and producing very high quality content, it's just that their cost structure was way to high.  At the time, Yahoo stepped in an offered to buy Rivals for $25M in cash but they proposed a incredibly complex structure designed to somehow keep the near term losses off of their books.  That deal fell through (like so many "rescue attempts" did in 2001) and the company was basically liquidated by the VCs.

Fast forward 6 years later and Yahoo is laying out $100M in cash for a newly revived Rivals.  From what I understand, the founder bought out all the assets in 2001 and restarted the company with a non-bubble cost structure and mentality.  Six years later he is rich man and Yahoo is out an additional $75M because they let a bad cost structure distract them from the reality of a great product.

For me, a deal like Rivals reinforces some great investment lessons:

1)  Even if good products are damaged by bad business decisions, they are still good products.
2)  The time to buy is when everyone else is selling.
3)  You can always reduce expense more.
4)  Don't give up on a good product too soon.

Congrats to any of the original Rivals team for sticking it out.  That's what being a passionate entrepreneur is all about.

YahooMail, Storage, and the Battle For Personal Data

Yahoo! announced today that it was effectively ending the long running game of e-mail storage one-up-manship between itself, Google and Microsoft by offering unlimited e-mail storage for free.  Not to be outdone, Google and Microsoft can be expected to follow suit shortly at which point Internet e-mail junkies will be in a universal state of infinite e-mail storage bliss.  This move by Yahoo highlights two key trends, one of which is obvious, the other not so obvious and represents just the tip of the iceberg in what is likely to be an avalanche of free storage offered by major Internet sites in the coming months.

Cheap Storage = Free Storage
The obvious trend highlighted by Yahoo's move is that storage is damn cheap and getting cheaper.  As I outlined in a recent piece on storage trends, in the last 15 years storage capacities have grown by a factor of almost 6,000 X while prices have declined by a facto of over 13,000X.  Yahoo's move to offer unlimited e-mail storage is basically an acknowledgement that storage costs have declined to the point where the incremental cost of providing more storage is more than offset by the incremental revenue opportunity. A similar game of free storage one-up-manship is also currently playing out in the online storage/backup space with sites such as AOL's XDrive, OneBox, and Streamload announcing ever higher levels of free storage (25 gigabytes is the current frontier).

Storage and the Webtop
However, beyond dirt cheap storage Yahoo's move highlights another trend which is just starting to play out.  This trend is the migration of desktop data to the online storage cloud and the fight between Google, Yahoo, and Microsoft to control this data because he who controls the data will most likely control the "webtop" and the suite of web-based applications that access that data.

What is the "webtop"?, well the webtop is basically a web-centric version of the desktop OS.  Microsoft's $270BN market valuation attests to the value of the desktop OS (and its suite of integrated applications) and there is a growing belief in the tech world that much of that $270BN may be up for grabs again as end-users make the platform transition from desktops to webtops.  There is also a realization that in the standards-based world of the web, the only real sustainable advantage is control of unique customer data because without that data one webtop OS is basically indistinguishable from the next.

To put this another way, if you think about what really makes your PC or laptop "your" computer it's not the desktop operating system, but all the unique data and applications you have installed on top of it.  If you take away that data all you have left is a terminal that most people could care less about.   On the web, there is no need to install applications because all applications are universally accessible.  That leaves personal data as the only unique asset that defines one webtop from another.

With this as the context, Yahoo's offer of unlimited storage starts to look a bit more strategic than just riding the curve of cheap storage, in that what it is really designed to do is to protect and extend their franchise of hosting what is arguably the most important personal data that most people have today: their e-mails.  Controlling this data makes it much more likely that Yahoo will not only be able to keep a large percentage of their users using not just YahooMail! but all of their other services that feed off of user data.  By building hooks for their various apps into users personal data, Yahoo will make using it's own applications the path of least Resistance.  For example: Want to share a photo you just got in the YahooMail!, click on the "Share via Flickr" button and you are all set.

The Tip of the Iceberg
So where do these two trends lead us?  A lot more free storage from the big three is one sure bet.  In fact, I would expect that by the end of 2008, Google, Microsoft, and Yahoo will basically offer unlimited storage for everything: e-mail, pictures, music, application data, you name it, for free.  They will do this not just because storage is cheap, but because they realize that by giving away personal data storage they have a much better shot of controlling the webtop and its constellation of tightly integrated web applications.

Let The Battle Begin
The battle for the webtop and the platform transition spoils that likely accompany it is far from over.  Yahoo and Google must still address serious gaps in virtualization, offline capability and application integration.  However, one of the most, if not the most important early battles in that war, the battle over the control of personal data, is now well and truly underway.

Top 10 Internet Stocks of 2006

While 2005's Internet winners were dominated by the themes of online advertising and gambling, 2006's key themes were China and broadband.  Four of the Top 10 stocks were focused on China while 3 were focused on broadband infrastructure.

To qualify for this list the company had to start 2006 with at least $50M in market cap and its business had to be focused on the Internet.  So, without further ado, here are the Top 10 Internet Stocks of 2005:

1. Navisite
   Price Change: 458% Ticker: NAVI
   Comment: If you need proof that hosting is hot again, look no further than Navisite.  This once left for dead application hosting company  had a huge year including  a suspiciously  strong December.
2. Gigamedia
   Price Change: 243% Ticker:GIGM
   Comment: What do you get when you combine China with gambling? An incredibly hot stock that's what.
3. China.com
   Price Change: 197% Ticker: CHINA
   Comment:  The right ticker at the right time.  With anything China related hotter than a  Szechuan Spicy  Beef, it was China.com's time to shine.  It didn't hurt that  they made a major move in online gaming too.
4. Akami
   Price Change: 167% Ticker: AKAM
   Comment:  Internet video was white hot in 2006 and Akami's CDN serves the most video of anyone so investors saw them as platform level pure play on the growth of internet video.
5. Knot.com
   Price Change: 129% Ticker: KNOT
   Comment: After a successful merger with long time competitor The Wedding Channel, the Knot.com proved that near monopoly positions in attractive internet content niches can be quite rewarding.
6. C-Trip
   Price Change: 117% Ticker: CTRP
   Comment: China's top online travel agency benefited from the overall China craze and increased travel by the emerging chinese middle class.
7. Omniture
   Price Change: 117% Ticker: OMTR
   Comment: The only IPO on the list.  Omniture didn't go public until mid-year but made up for its late start as it became the SASS darling of online analytics and took market share quickly.
8. The9
   Price Change: 111% Ticker: NCTY
   Comment: China.  World of Warcraft.  Need I say more?
9. Internet Gold
   Price Change: 100% Ticker: IGLD
   Comment: Israeli ISP rises due to merger with rival and ventures in online video and broadband access.
10. Priceline
    Price Change: 99% Ticker: PCLN
    Comment: Rising hotel and airline fares combined with increasing business travel send consumers scrambling to find cheap deals the Internet.

Guess What? Internet Stocks Had A Terrible 2006

Lost amidst all the Web 2.0 hype, the eye popping deals like Google's $1.65BN for 18 month old YouTube and the huge increases in VC Internet-related investments is a open secret which many people in Silicon Valley don't seem to have fully assimilated: Internet stocks had a pretty terrible year in 2006.

While all the major indexes had decent years with the Dow up 16.3%, the S&P 13.6% and the Nasdaq up 9.5%, the Internet sector actually turned in a return of negative -2.8%.  Now that doesn't include dividends, but few if any Internet stocks pay dividends so I think it's safe to say that Internet stocks, as a whole actually lost money in 2006.

"Just how is that possible?" you might ask, well you need to look no further than the 5 largest Internet Stocks.

The table below lists the Top 5 Internet Stocks in terms of market cap:

                                                           

Ticker	Cap$BN	1/1	12/31	%Chg
GOOG	141	414.86	460.48	+11%
EBAY	42	43.22	30.07	-30%
YHOO	35	39.18	25.54	-35%
SCH	24	19.34	14.67	32%
AMZN	16	47.15	39.46	-16%

In aggregate the market cap of the Top 5 Internet companies dropped $18.5BN or 6.7%.   That's a huge hole for the rest of the sector to try and make up (which it didn't).  Matters weren't helped by the complete implosion of the online gambling sector either which lost well over half its value.  And don't try to blame it all on Yahoo and Ebay either, even small cap internet stocks had a bad year.  While they didn't lose money, they gained a paltry 2.6% compared to the Nasdaq's 9.5% return.  Even the Net's new standard bearer, Google, underperformed the market, a fact which could probably win you more than a few bar bets this week. 

So don't let all the hype fool you, the Internet remains a pretty treacherous corner of the market no matter how much AJAX and Adsense you throw at it.

The Storage Explosion

I am a big believer in the “scarcity and abundance” theory of IT development. The theory basically postulates that if you want to understand the near term future of information technology development the most important thing to consider is the scarcity and/or abundance of the “big 3” foundations of computing power: processing power, storage and bandwidth. Understanding the absolute and relative levels of these three technologies is the closest thing possible to having an IT industry crystal ball as they have a huge influence over everything from system architectures to capital investment plans to end user demand.

A 15 Year Rocket Ship Ride
It is with this in mind that I found a recent article on Tom’s Hardware to be fascinating. The article details the increases in storage capacity and performance over the last 15 years. Some of the numbers involved are mind boggling.  In the last 15 years, the storage capacity of top-end hard drives has increased 5,907X from 130MB in 1991 to 750GB in 2006. To put this in perspective, during the same time period here are some increases in the other core technologies of a “bleeding edge” PC:

                                       

End User Technology	1991	2006	X Increase
LAN Bandwidth	10 mbit/s (10BaseT)	1 gbit/s (1000BaseT)	102X
WAN Bandwidth	14.4 kbit/s (v.32bis)	3 mbit/s (Cable)	213X
CPU Performance	54 MIPS (486DX)	27079 MIPS (X6800)	501X
Hard Disk	130MB	750GB	5907X

The price difference is even more dramatic. In 1991 a megabyte of storage cost about $7.00, now it costs $0.000527. That’s a 13,274X price improvement or a 99.9925% price drop in 15 years. Not too shabby.

It’s interesting to note though that disk I/O performance improvement has lagged dramatically with only a 121X improvement from 0.7 MB/s to 85 MB/s. This clearly makes disk I/O one of the biggest, if not the biggest bottleneck in a modern PC. Put another way, it would theoretically take 3.1 minutes to write the entire contents of a 130MB  disk drive in 1991 while it takes about 2.5 hours to write the contents of a 750GB drive in 2006 (as anyone who has tried to back-up a monster like this well knows).

Forecast Calls For: More, Cheaper Storage
The near term prospects for additional gains in storage capacity remain promising. For traditional magnetic storage hard drives, the introduction of perpendicular recording should continue to drive platter densities higher, but much like CPUs, the limits of physics are starting to catch up with magnetic hard drives which suggests that they will not be able to continue their capacity gains indefinitely. Not to worry though, two new storage technologies are finally coming to market this year including Flash memory hard drives and holographic storage. Flash memory hard drives use NAND chips instead of magnetic platters to store data. They have a number of important advantages over traditional magnetic media including dramatically faster file access times and the ability to process far more I/O operations per second.  For example, SanDisk just announced today a 32GB flash drive that accesses files in 0.12 milliseconds vs. 8 milliseconds for the best magnetic drive, a 67X improvement (although its read rate is only 64 MB/s).  Flash drives also draw 50-70% less power than magnetic drives, weight about 50% less, do not produce noise or vibrations, give off very little heat, and are less fragile than magnetic drives. But all these improvements do not come cheaply as the initial 32GB flash drives will cost more than $20/GB or about 35-40X more than a magnetic drive.  Still you should expect to these as an option on high end ultra-portable laptops in a few months.  I also expect that many PC enthusiasts may buy a flash drive for their "C:" drive to dramatically improve Windows Vista boot times.

2007 will also see the first commercially available holographic storage drives, although these drives are likely to only compete with archival media, such as tape drives and Blu-Ray media, for the foreseeable future. Holographic areal storage densities have doubled in just the past year and at 515Gb/inch are already more than 2X those of the most advanced magnetic drives.  To put this in perspective, Sony’s Blu-Ray disks which have just started shipping after many delays have a 50GB capacity while InPhase’s initial holographic disk (which had it’s 1^st shipments just a few days ago) have a 300GB capacity and should go to 500GB in the next year or so. Put another way, you could store 106 DVDs on a single (slightly larger) holographic disk (although you can only write to these disks at about 23 MB/s, so be prepared to spend over 6 hours putting you DVD collection on a single holographic disk). Because it uses light rather than magnetic heads to access data, holographic storage should theoretically delivery better access times and faster read throughput than magnetic storage mediums. Holographic storage is expected to seriously challenge tape backup systems in the short term, but could also displace consumer storage mediums such as DVD and Blu-Ray as prices fall. Holographic disk drives are theoretically possible, but significant technical challenges still have to be overcome especially in terms of being able to rewrite the media and improve write speeds.

What Happens In A World Awash With Storage?
You may be asking yourself, this is all very interesting but why do you, someone who generally is focused on software and the internet, care so much about storage? I care because storage capacity is clearly the most abundant and faster growing component of the “big three”(processing power, bandwidth, and storage) today and it looks as though the relative disparity between the three may even increase further in the short term.  This situation should have some very interesting implications for both software and internet related businesses, some of which I will outline in my next post on the topic.

Search + State + Metadata = A Search Application

The major search engines have all built incredibly impressive and expensive infrastructures with one main goal in mind:  finding the one result or small set of results that enable a user to find the specific information they are looking for amidst the vast forest of information that is the Web.  Finding the specific tree in this vast forest, and finding it fast, is thus not only the benchmark by which all search engines are measured but also arguably the dominate axis of investment and innovation within the search industry today.

If you don’t believe me, just look at all the search start-ups that have launched in the last few  weeks including  Riya and Powerset.  While these start-ups each have a different focus, their main value proposition is the same: we help you find you are looking for faster and better than the other guys.

As I outlined in my last post though, the huge investments required to even have a prayer of “building a better Google” dictates that most “tree focused” search start-ups will either: A)  Be forced to concentrate on a small niche, B) Be acquired by one the one of the majors for their technology, or C) Be crushed by the major players when they add similar functionality.

Seeing The Forest for The Trees
The real green field in search today is not trying to find the tree, it is in trying to understand the forest.    The forest, is the vast interconnected sea of web of sites that make up the web and the flows of information that constantly course through it.  Once you can see this forest and observe how it changes over time, you can begin to derive insights and information that simply are not possible to discern with a single query.

To understand the forest you need three main things:  persistent search, state, and metadata.  Persistent search is simply a search query that is constantly running.  State is information about the state (what time is it, what things are connected to what, etc.) of the web each time a query is run.  And metadata is information that is derived from the set of search results returned for a particular query (how many results, what type of results,  who authored the results, statistical analysis of elements within the results, etc.).

Marrying these three items together adds two key dimensions to search:  time and context.   Time is an incredibly important dimension because once you can compare things over time, you can determine change and the rate of change.   For a primitive example of how to marry search and time, look no further than a little noticed (and very immature) feature on Google Finance.   If you look at the news for a particular stock in Google Finance, on the top left hand side you will see a little graph of the number of articles mentioning this stock over time.

Right now this feature is very novel (and it lacks state), but if you think about a more robust version of this that attempts to relate the number of articles to other times series, such as stock prices or trading volume, it starts to get much more interesting.

In terms of context, context allows a user to easily understand how one data point fits into a broader picture.  For example, at Vast.com, if you search for a specific used car, say an Audi S4, on the left hand side of the page you not only see all of the Audi S4’s for sale, but also the median mileage and median price of all those cars.  This metadata (the medians ) is not found in the search results, but produced by analyzing the aggregate results.

Now imagine charting this metadata over time (via a persistent query that captures the state each time it runs)  and you can easily envision a chart of the average asking price for a specific type of Audi S4 over time.  (Kind of like what mpire is doing with its auction search today.)  A marketing person at Audi (or its competitor) might be very interested to see how these values change over time or respond to price cuts for new cars or new model introductions.   Similarly a car insurance company might be very interested in this data to assess replacement values.   Kelly BlueBook might be interested in this data because it has the potential to seriously disrupt their business.

The Metadata’s the Limit
Things get even more interesting when you add in advanced text analysis techniques, such as natural language processing and entity extraction, to generate additional metadata.  All this metadata can then be correlated with not only data from other searches but with traditional structured data.  Want to understand if good or bad product reviews on blogs impact sales?  Search away.  Want to understand if a sharp spike in message board traffic is likely to impact a stock?  Search away.

Of course,  you won’t be able to simply type in a search to Google (or any other search engine for that matter) and find out the answers to such questions.  You will likely get you answers from purpose built “search applications”.  Who will build those applications? Mostly start-ups to begin with (many already are), but eventually established firms will also enter this space.  Expect to see a heavy focus on financial services and marketing related applications early on, but the concept has applicability across just about everything.   It will be very interesting to see just what is built and what succeeds.

Persistent Search: Search’s Next Big Battleground

What do you get when you marry ping servers and RSS with stored queries?  A whole new type of search that is destined to become the search industry’s next big battleground: Persistent Search.  While Persistent Search presents search companies with difficult new technical challenges and potentially higher infrastructure costs, it also gives them powerful mechanisms for building much stronger user relationships which may increase the value of their advertising services.

The Search That Never Stops
Simply put, Persistent Search allows users to enter a search query just once and then receive constant, near real-time, automatic updates whenever new content that meets their search criteria is published on the web.   For example, let’s say you are a stock trader and you want to know whenever one of the stocks in your portfolio is mentioned on the web.  By using a persistent search query, you can be assured that you will receive a real-time notification whenever one of your stocks is mentioned.  Or perhaps you are a teenager who is a rabid fan of a rock group.  Wouldn’t it be nice to have a constant stream of updates on band gossip, upcoming concerts, and new albums flowing to your mobile phone?  Or maybe you are just looking to rent the perfect apartment or buy a specific antique.  Wouldn’t it be nice to get notified as soon as new items which roughly matched your criteria were listed on the web so that you were able to respond before someone else beat you to the punch?  Persistent search makes all of this possible for end users with very little incremental effort.

Something Old, Something New
While the technical infrastructure required for Persistent Search services leverages existing search technology, there are several new elements that must be added to existing technology to make Persistent Search a reality.  These elements include:

1. Ping Servers:  Most blogs and an increasing number of other sites now send special “pings” to so called “Ping Servers” every time they publish new content.  The ping servers do things such as cue crawlers at a search engine to re-index a site or provide a summarized list of recently published information to other web sites.  Because ping servers are the first to know about newly published content, they are critical to enabling the real-time nature of Persistent Search.
2. RSS:  RSS feeds can be used both to feed raw information into Persistent Search platforms (in a similar fashion to what GoogleBase does) as well as to take processed queries out. RSS is a polling based mechanism so it does not provide real time notification, but it is good enough in most cases.
3. Stored Queries:  Stored queries are simply search queries that are “saved” for future use.  Ideally, the stored query is constantly running in the background and it flags any new piece of content that meets the search criteria.  While this is a simple concept, it presents some very difficult technical challenges.  The easiest way for a search engine to do a stored query would be to execute the stored query into its existing index at some regular interval, say once an hour.   However, executing each unique stored query 24 times a day could start to become very expensive if Persistent Search starts to take off.  One could easily imagine search companies in the near future executing billions of incremental stored queries an hour.  Processing these added queries will take lots of extra resources, but will not generate the same amount of revenue that traditional ad-hoc search queries generate because stored queries will often return no new results.  One alternative would be for search companies to use the “stream database” query techniques pioneered by start-ups such as StreamBase.  These techniques would allow them to query the new content as it flows into their index, not just reducing overall query load but also improving latency.  However changing their query approach is a huge step for existing search companies and therefore one that it unlikely to be undertaken.  One more likely approach might to use special algorithms to combine stored queries into “master queries”.  This reduces the number of queries that need to be executed and uses simple post-query filters to “personalize” the results.  Given their critical important to overall quality of persistent search, the design of “stored query architectures” is likely to become one of the key technical battle grounds of search companies, much the way query result relevancy has been for the past few years.

Once these three pieces are put together, search companies will be in position to provide rich Persistent Search services.  The results of those services will be distributed to end users via e-mail, RSS, IM, SMS, or some pub-sub standard, depending on their preferences and priorities.

The Business of Persistent Search
From a business and competitive perspective, Persistent Search has a number of very attractive aspects to it relative to traditional ad-hoc queries.  Traditional ad-hoc search queries tend to result in very tenuous user relationships with each new query theoretically a competitive “jump ball”.  Indeed, the history of search companies, with no less than 5 separate search “leaders” in 10 years, suggests that search users are not very loyal.

Persistent Search presents search companies with the opportunity to build rich, persistent relationships with their users.  The search engine that captures a user’s persistent searches will not only have regular, automatic exposure to that user, but they will be able to build a much better understanding of the unique needs and interests of that user which should theoretically enable them to sell more relevant ads and services at higher prices.   They will also stand a much better chance of capturing all or most of that users’ ad-hoc queries because they will already be in regular contact with the user.

It is this opportunity to build a long term, rich relationship directly with a uniquely identifiable consumer that will make persistent search such an important battle ground between the major search companies.   Persistent search may be especially important to Google and Yahoo as they attempt to fight Microsoft’s efforts to imbed MSN Search into Windows Vista.

It should also be noted that enterprise-based Persistent Search offers corporations the opportunity to improve both internal and external communications.  For example, it’s not hard to imagine the major media companies offering persistent search services or “channels” to consumers for their favorite actor, author, or singer.

The State of Play
As it stands, Persistent Search is in its infancy.   The world leader in persistent search is most likely the US government, specifically the NSA, however the extent of their capabilities appears to be a closely guarded secret.  Some of the commercial players in the space include:

• Pub-Sub is in many ways the commercial pioneer of the space.  It operates one of the largest ping servers and has been offering customized key word-based “real time” persistent searches for some time.  Some start-ups are even building vertical persistent search services on top of pub-sub’s infrastructure.  However Pub-Sub lacks the infrastructure and resources of a complete search engine and lacks the consumer awareness and multi-channel distribution capabilities needed to reach critical consumer mass.
• Google itself offers Google Alerts, a service that enables users to be e-mailed the results of stored queries, however this service relies on Google’s general index and so it could take days for new content to appear.  It also does not appear to directly incorporate ping servers and does not offer any distribution mechanisms beyond e-mail.  In addition, as a beta service, it’s not clear that it is capable of scaling efficiently should demand take off.
• Real Time Matrix is a start-up focused on aggregating RSS feeds and re-broadcasting them based on user preferences.  The net effect of their technology is to deliver real-time Persistent Search-based RSS feeds to consumers.
• Technorati, a popular blog search engine, allows users to create “watchlists” of persistent searches.  However Technorati limits its index to blogs and so does not offer a comprehensive search service.
• Yahoo and MSN offer their own alert services and while these services provide very good distribution options (e-mail, IM, SMS), they do not integrate into their search engines but just offer canned updates for things such as news stories and the weather.
• WebSite-Watcher and Trackengine offer services that track specific sites and/or RSS feeds for changes, but they do not allow fine grained free text queries and are just focused at the site level.

Despite this activity, no one has yet to put together an end-to-end Persistent Search offering that enables consumer-friendly, comprehensive, real-time, automatic updates across multiple distribution channels at a viable cost.   That said, the opportunity is clear and the competitive pressures are real, so I expect to see rapid progress towards this goal in the near future.   It will be interesting to see how it plays out.

Real Estate, Cars, Jobs: Watch Out World, Google Base Has Only Begun To Stir

There has been a lot of commentary about Google Real Estate’s beta launch earlier this week.  It turns out that Google also is quietly testing a similar service for cars and jobs as well.  Both the real estate launch and the car launch take data from Google base and integrate it with Google Maps, providing a consumer friendly front end to the database.  (My guess is that the appearance of both services this week probably has something to do with the release of Google Maps’ 2.0 API).

With the launch of these Google Base front-ends, Google is clearly putting into place the major pieces required to support its vertical search platform.  Broadly speaking, such a platform requires 4 major pieces:

1. A big, highly scalable database that can handle lots of queries.  This, of course, is what Google Base was all about.
2. Consumer friendly front ends to access these databases.  The auto and real estate front ends are obviously the first of such front ends.
3. A large, robust, crawling farm.  This is obviously Google’s crown jewel.
4. A set of intelligent algorithms to find, classify, and flag listings.  We have yet to see this from Google.

Most people remain unimpressed by Google Base because it doesn’t seem to contain a lot data.  That’s because what you are seeing is a work in progress that is being purposely hobbled to reduce load during the testing phase.   Google has now built beta versions of pieces #1 and #2.  We will un doubtedly soon see pieces #3 and #4.  Only when those pieces are in place will Google Base fulfill its potential.

In terms of piece #3, Google likely has to make changes and updates to its core crawler code in order to accomplish this.  This is a non-trivial task and not something undertaken lightly.  Piece #4 requires a decent number of Google’s PhD’s to build and test algorithms for recognizing listings within unstructured data and then structuring it, also a non-trivial task.  However, as someone who watched Google hire some of the brightest minds in unstructured data management I can tell you that have more than enough firepower to accomplish the mission.

Once Google, hooks up pieces #3 and #4 (likely at the same time) a flood of information will cascade into Google Base and from there into the fronts that they recently launched.  If you want to view a good approximation of what Google Base will look like once it is finished, go look at Vast.com. 

Losers and Bigger Losers
There will be two sets of losers in all of this.  The first and most immediate set of losers will be the start-up vertical search players (indeed one can only imagine the long faces at Trulia (and their VC backers) when they got their first look at Google Real Estate).  Of course losers may not be an accurate term as the correct response to Google Base from these companies should be to pick up the phone and call Yahoo, Microsoft, IAC, and AOL and say “you guys need to buy us because Google is going to clean your clock” and who knows some of the big boys might just hit the panic button and write a few big checks.

The problem for these vertical search players is that Google has set a very high bar by integrating its vertical search seamlessly into it’s free text query engine, crawler farm, and data base.  If a Yahoo or Microsoft, were to buy several different vertical search start-ups to respond to Google Base (and they must respond one way or the other) they would inherit a huge integration headache and be faced with a massive back-end restructuring.  Faced with this headache, some of them may well decide to follow in Google’s footsteps and built it from scratch. Alternatively, they may prefer to acquire a more “horizontal” vertical search play, such as Vast, which has already built a multi-vertical crawl and classify engine.  Either way, if I were running a vertical search engine I would be putting a sign on my front door right about now reading “No reasonable offers refused”.

The second set of losers in this are the well established listings-focused Walled Gardens of the Internet.  As I have outlined before in detail, these Walled Galled face a fundamental threat from search.  A fully functioning Google Base will make that threat more real than ever.

I don’t know when pieces #3 and #4 will be launched, but for me that will be the single most interesting day in the short life of Google Base and far more deserving of hoopla than the launch of a few simple front ends.