The Internet Gold Rush of the late twentith century created an environment of experiemental business models, some of which were pretty wacky. The flood of venture capital often allowed the underlying weaknesses of those business models to be hidden from the management of those dubious enterprises. As CIO of O’Reilly C.J. Rayhill says, when the swamp is full, the tree stumps are covered. In today’s VC-poor atmosphere, the water is low and the Internet swamp is littered with stumps.

Yahoo Seriously Troubled

Of all the Internet businesses to emerge from those salad VC days, perhaps the most useful one for the entire Internet community is the search engine. Where would the average Internet surfer be without the humble Yahoo, Google or Lycos? Without search engines, the Internet revolution would have been merely a footnote in Computer Science courses.

Almost all of the large search engines allowed anyone to add his own link to their database. Of course, each search engine had software robots called spiders to extract the hyperlinks from a given web page and follow them. Thus, for each human submitted page an exponential number of new pages might be discovered.

Not only could one submit links for free, but users could also search the link database for free. Search engines usually depended on banner ads for revenue. Although this seemed to work for a time, banner ads are clearly not going to be the source for unbriddled revenue growth. How can the large search engines survive?

What Businesses Need

When I worked for a small Needham, Massachusetts company called CareerSearch, I spent most of my time mapping databases from book publishers of industry contact information into CareerSearch’s own database system. CareerSearch is a subscription-based web application that targets the very small market of out-placement firms and university career centers. CareerSearch’s charter is to service the needs of other businesses. They do not collect their own data, but they do present a robust interface into an amalgamated database of over a million companies. Most importantly, they are a profitable company and they have never used banner ads. Instead, customers need to buy rather expensive subscriptions to access the CareerSearch application. CareerSearch exists because its data suppliers, those book publishers of contact information, couldn’t or wouldn’t make their data available in a format that other businesses wanted.

What does this have to do with Internet search engines?

The problem with the business model of search engines is that it depends on casual viewers for revenue. This is a huge mistake. Search engines need the users to supply URLs for their database. In fact, search engines should consider paying users to add verifiably correct links to their systems, because here’s the secret that these companies don’t seem to understand: the value of a search engine is its link database.

Search engines seemed to have conceived of only one use and only one kind of customer for their data. The question search engine companies should be asking is, how can we sell our data to businesses? There is a wealth of information locked away in the link databases of Yahoo and Google. Remember, each URL in their system is linked to a series of keywords (at a minimium). There are so many uses for this data that companies have to build web spiders of their own just to get at the information. Clearly, there is a need for a richer interface into these systems.

Making Link Databases Available

Once a search engine decides to service business clients, the next question what sort of interface is most apropriate? For example, should Google build a subscription-only web site with a radically different layout that allows for more robust searching options? O’Reilly Senior Software Engineer Tim Allwine and I think this is the wrong way to go. Businesses know what they want and they are likely to have programmers who can build applications that manipulate the link data in a relevant way for them. Search engines need to build subscription-based Web Service interfaces that provide rich querying capabilities and easy data retrieval. Remember, Web Services (like SOAP and XML-RPC) are designed for programs, not for people. Each business client will need to build an application that makes the search engine Web Service fit its particular business need.

For example, XYZ Corp wants to measure how effective its press releases are. One way to do this is to see how often the content from that document gets quoted in online media. XYZ Corp would love an interface that allows them to type in a unique sentence from the press release and get the number of sites that use that phrase. XYZ Corp can expand on this simple idea by creating an internal tool that automatically looked for press releases and charted the citations over time. If search engines provided Web Services interfaces, this type of application would be very easy to create. By supporting IT infrastructure, search engines would have a business model that makes sense.

Web Services can be very easy to build. The kind that search engines would build are particularly easy because they would be read-only (i.e. clients aren’t modifying the link database) and non-transactional. By helping businesses help themselves with Web Services interfaces, large search engines get a license to print money, which is exactly what places Yahoo, Lycos and Google are looking for most right now.

[Original post and comments.]