A Search Engine is a tool or device used to find relevant information. Search engines consist of a spider, an index, relevancy algorithms, and search results.

Also known as a web crawler or robot, a search engine spider is an automated program that reads web pages and follows any links to other pages within the site. This is often referred to as a site being "spidered" or "crawled". As the spider goes through these pages, it records the words it finds on the pages. It makes notes about how many times each word appears, whether the words are weighted in certain ways, perhaps based on size, location, or HTML markup, and decides how relevant the words are based on the links that come in to the page, and on the general context of the page.

The index is a collection of data used as bank to search through to find a match to a user fed query. The larger search engines have billions of documents in their catalogs. When search engines search they search via reverse indexes by words and return results based on matching relevancy vectors.

Spiders start their journeys with a list of page URLs that have previously been added to their index (database). As it visits these pages, crawling the code and copy, it adds new pages (links) that it finds on the page to its index. The spider returns to the sites in its index on a regular basis, scanning for any changes. How often the spider returns is up to the search engines to decide. Website owners do have some control in how often a spider visits their site by making use of a robot.txt file. Search engines first look for this file before crawling a page further.

An index is like a giant catalogue or inventory of websites containing a copy of every web page and file that the spider finds. If a web page changes, this catalogue is updated with the new information. To give you an idea of the size of these indexes, the latest figure released by Google is 8 billion pages.

It sometimes takes a while for new pages or changes that the spider finds to be added to its index. Thus, a web page may have been "spidered" but not yet "indexed." Until a page is indexed - added to the index - spidered pages will not be available to those searching with the search engine.

Search engines then must weight the value of each page for the words that appear on it. This is the trickiest part of what a search engine has to do, but also the most important. At the most simple level a search engine could simply keep track of every word on the page, and record that page as relevant for searches with that keyword. This wouldn’t do much good for most users, however, as what is desired is the most relevant page for their search query.
For more information check out the Beginner's Guide to SEO.

Investing wisely!

Regards,

Your Staff and Stan Krol/CEO