The centerpiece of this chapter is the handy-dandy list of top-flight search
sites and systems, some of which you may never have heard of. Trust me;
they’re either important search sites or feed important sites or many smaller sites. In case you missed it, here are these sites and systems more or less in order of importance:

Google

Yahoo!

The Open Directory Project
Inktomi

Teoma/Ask Jeeves
AltaVista.com

FAST/AlltheWeb
Zeal

Why is the Open Directory Project — not exactly a household name — so high on the list, higher than some of the top search sites? The Open Directory Project feeds Google, AOL, Ask Jeeves, Netscape, EarthLink, Lycos, and many more. Also, Inktomi is high on the list because it feeds MSN, which is the world’s fourth most important search system.

This list is likely to change soon. My prescient predictions are summarized in the following list:

Yahoo! dumps Google. This event won’t change the list, but it will change

the importance of several players on the list. At the time of writing, Yahoo!

has already started dumping Google, and will probably complete the

process by the middle of 2004. Most of the search results at Yahoo.com

currently come from Google. But Yahoo! now owns Inktomi, AltaVista, and

the immense FAST/AlltheWeb index.

Yahoo! has begun experimenting with using Inktomi results in some of its

non-U.S. search sites, and by the time you read this, Yahoo! will probably

have completed the process and be running its own search indexes. The

results? From the perspective of someone trying to get traffic to a Web

site, Google becomes a little less important (though still very important),

and Inktomi becomes far more important because it will be feeding Yahoo!

search results. FAST/AlltheWeb may also be part of the equation, making

that search system much more important.

Here’s a very likely scenario. Yahoo! merges in some way with the Inktomi

and AlltheWeb indexes. It dumps the (much smaller) AltaVista index, but

keeps the AltaVista brand name, which is more widely recognized than

the other two.

21

MSN dumps Inktomi. Currently, MSN.com doesn’t appear on the list

because its search results come from Inktomi and LookSmart (which in
turn is fed by Zeal). But Microsoft has begun building its own index and
will soon be in the search engine business, which is too important for
Microsoft to ignore. MSNBot, as it’s known, is traveling around the Web
as you read this. It’s quite likely that MSN will be a hugely important
search site with its own search system feeding results.

The result? The list of important search sites and search systems has now expanded by one very important system. Furthermore, Microsoft owns
five of the world’s top 17 Web sites (MSN.com, Passport.com, Microsoft.
com, MSNBC.com, and Hotmail.com), all of which will be fed search
results from the new Microsoft system. And when the next version of
Windows ships in 2006, the search engine will be built into the operating
system — you’ll be able to search your hard disk for files and e-mails,
search for the content within spreadsheets and documents, and search
the Web, all from the same place.

Grub comes alive. Grub (www.grub.org) is an interesting project

that may, one day, be very important. Owned by LookSmart, Grub

plans to build a massive index of billions of pages by using distributed computing — people around the world using their spare computing downtime to crawl the Web for Grub. I don’t know if this undertaking will become important, but it’s certainly one to watch.

Nutch takes off. I think Nutch (www.nutch.org) may also become impor­
tant at some point. It’s an open-source project that plans to create a huge,
multibillion-page index and make it available to anyone who wants it. At
the time of writing, it’s in the early stages, but it has the backing of some
important people and may turn into something significant. On the other
hand, it may not.

Visit www.SearchEngineBulletin.com for the latest news on the Yahoo!,

MSN, Grub, and Nutch developments. I’ll let you know how to get indexed in the appropriate places.