Facts, Fiction and views by valuepitchers

Google, Yahoo! and the Microsoft Corporation are the three leaders in the search engine industry. They use crawlers to find their algorithmic search result pages. It is not necessary to submit the pages that are linked from the other indexed pages of search engines, as these can be found automatically.



Yahoo! and some other search engines either charge a pre-determined fee or the cost per click (CPC) for their paid crawling and submission services, but they guarantee a fairly good service. These programs usually guarantee being included in the database, but they do not guarantee a specific rank within the results on the search engine pages.



Critics have come down heavily upon the paid inclusion program launched by Yahoo! These critics not only include the advertisers, but also the competitors. The Yahoo Directory and the Open Directory Project are two major directories which require manual submission. The editorial review is not automatic either.



It is possible to create an XML-type of feed for Google Sitemaps, which is offered by Google, and submit it free of cost to make sure that all of the pages of search results are found, especially those pages that cannot be discovered easily (that is, by automatically clicking on the links provided).



The crawlers employed by search engines may look at various factors while crawling a website. The search engine does not index every page of search results. One of the main factors that determines whether a page would get crawled or not is the distance between the page and the root directory of the website.