Finding your way through Search Engines.

Google, Overture,Yahoo, Altavista – these are just a few of the many search engines on the internet today. Search Engines are the most visited websites on the internet with most internet users utilising them as an entry point to find other websites.

What is a Search Engine?

A tool that enables users to locate information on the World Wide Web. Search engines use keywords entered by users to find Web sites which contain the information sought.

Types of Search Engines

Spider/crawler based
Crawler-based search engines have three major elements. First is the spider, also called the crawler. The spider visits a web page, reads it, and then follows links to other pages within the site. This is what it means when someone refers to a site being "spidered" or "crawled." The spider returns to the site on a regular basis, such as every month or two, to look for changes.

Everything the spider finds goes into the second part of the search engine, the index. The index, sometimes called the catalog, is like a giant book containing a copy of every web page that the spider finds. If a web page changes, then this book is updated with new information.

Sometimes it can take a while for new pages or changes that the spider finds to be added to the index. Thus, a web page may have been "spidered" but not yet "indexed." Until it is indexed -- added to the index -- it is not available to those searching with the search engine.

Search engine software is the third part of a search engine. This is the program that sifts through the millions of pages recorded in the index to find matches to a search and rank them in order of what it believes is most relevant.
EG. Google, MSN

Directories
A human-powered directory, such as the Open Directory, depends on humans for its listings. You submit a short description to the directory for your entire site, or editors write one for sites they review. A search looks for matches only in the descriptions submitted.

Changing your web pages has no effect on your listing. Things that are useful for improving a listing with a search engine have nothing to do with improving a listing in a directory. The only exception is that a good site, with good content, might be more likely to get reviewed for free than a poor site.

Paid placements and paid inclusion
Search results are definitely becoming confusing for the end-user. Unless clearly marked, the searcher has no idea if they are receiving a result that has been paid to be ranked higher or one that has been ranked by a crawler based engine's un-biased algorithms.

Some search engines, such as Lycos and Google, very clearly identify which results are paid for and which are not.

What is the difference between paid placement and paid inclusion?

A paid placement engine requires the advertiser to "bid" in an auction environment to attain a specific position for each keyword (some constraints may apply like Google's click-through performance rating.)

An XML paid inclusion engine does not guarantee any position, however, does guarantee that the most important pages on your site are in the search engine database. Both static HTML and dynamic (database driven) pages can be converted into XML and therefore included. While there is no position guarantee, when your listing does appear chances are it was a great fit for your page.