Search Engines

Search Engines

A search engine is basically an online application which web surfers use to find web sites and documents using a query of normally two or three keywords. The best known examples of these are Google, AltaVista, Fast/AlltheWeb and Teoma, although Google is by far the most popular. One that the general public may not know but is none-the-less important is Inktomi, which provides the search results for several Web Portals, notably MSN and up until a couple of years of ago also powered the secondary results of Yahoo.

Search engines are defined by the way they build their databases. A search engine runs free crawling applications which are programmed to start at a defined point on on the web, cache the data it finds, and then follow links to other documents to do the same. These applications are commonly known as robots or spiders and can be seen in your raw log files under a variety of user agents or identities. Once the spider has finished its crawl, the data is measured against the search engines unique and secret algorithm and each document is compared and scored.*

*This is a generalised definition. SEs operate in different ways and exact methods are known only to specific SE employees.

For more information on how to improve your website for high rankings in search engines take a look at our section on site optimization .

So Who Are The Major Players?

OK. First thing to remember is that many SE's are partnered with and often rebranded in many favourite Web Portals. For instance the backbone for the BBCi Search Engine is powered by Google. So the reach of some engines is huge even though web surfers may not necessarily directly search from the SE's home page.

Teoma is one of new boys in search technology "Teoma was founded in 2000 in Piscataway, New Jersey by a team of scientists from Rutgers University. Teoma means "expert" in Gaelic. Ask Jeeves, Inc. acquired Teoma in September 2001. "

Teoma is an excellent search engine with the potential to become as popular as Google. It has a free crawling spider and it's index is updated fairly regularly. There is also a PFI service if you want to get your site listed quickly and spidered regularly. More details of this, see resources opposite.

Teoma now powers the search results of Ask Jeeves. Teoma's spider can be found in your raw logs under the user agent of "TeomaAgent".

Google was launched in 1998 by Stanford University graduate students Larry Page and Sergey Brin. 4 years on it has become without doubt the most popular search service on the Internet and drives more search based traffic than any other website, approx. 150M searches per day. Some webmasters have reported over 90% of their traffic as generated by Google even when their site is equally well listed elsewhere.

Google powers search for many Web Portals, notably AOL, NetScape, Yahoo and LookSmart UK. Its spider goes under the user agent identity of "Googlebot".

Fast is a Norwegian based search company that has quickly built a name for itself. It first came to being in 1999 and claims one of the biggest databases of web documents and as the name suggests is incredibly fast. The search engine is built on very strong and reliable technology and search results are generally excellent. Fast's figurehead site is AlltheWeb which is not well known by the general public. Although erratic in recent times Fast spiders do crawl the web, but if you want guaranteed inclusion, Fast's PFI service will list you in 2 days and your pages spidered every 48 hrs.

Fast powers the search results of much of the Lycos network.

Inktomi was one of the pioneers of search technology. Although not a search engine per se, it powers the results for a number portals, notably MSN and Overture. Inktomi's database is built on a combination of paid and free listings. Paid listings are via partner sites such as Position Technologies and free crawling using its spider "Slurp".

Inktomi looked like they were pulling out of the search service industry a while ago when they lost their contract with Yahoo which quickly led to further losses with other partners. However their partnership with MSN still appears to be strong and with a 2 week update cycle and rumors they are chasing new contracts, Inktomi are looking stronger than they have done in the past few years...

All Rights Reserved

Our Services | FAQ | Contact Details

Build | Optimise | Promote | Earn | News | Home

:: Directories :: What are they and who are they. What are the benefits and the pitfalls.
:: Search Engines :: Who are the major players and how do you get listed in them. What partnerships do they have.
:: Related Portals :: Getting related sites, guides and directories to link to you.

Resources :





MSN - Search

Fast - AlltheWeb

Position Technologies

BBC Search