Facts, Fiction and views by valuepitchers

The optimisation of websites for search engines by specialists known as webmasters and providers of content dates back to the mid-1990's. The first search engines catalogued the World Wide Web in its early days.



In those days, a webmaster simply had to submit a page or a URL (uniform resource locator) to the various search engines. The search engines would then send a spider to crawl down the page containing the search results and extract links from it to other pages. The information returned would then appear on the page and be indexed.



The process of search engine optimisation involved downloading a page (which was done by a search engine spider) and storing it on the server of the search engine. A second program, called the indexer, would then extract information about the page (such as the words contained on it, their location, the weightage given to specific words and the links contained on the page) and place into a scheduler to be crawled at a date in the future.



The owners of websites began to understand the importance of the system of ranking of websites (that is, they began to realise that websites that were ranked higher were more clearly visible on the pages of search engine results). As a result of this, opportunities for both White Hat search engine optimisation as well as Black Hat search engine optimisation professionals were created.



According to Danny Sullivan, an expert in the field of search engine optimisation, the term search engine optimisation was first used by Usenet, who sent a spam message on the twenty-sixth day of the month of July in the year 1997.



The search algorithms used in the early days of search engine optimisation were heavily dependent upon the information provided by the webmaster. This included such keywords as meta tag and the index files that appeared in such search engines as ALIWEB.



Meta tags provided the guidelines to the content on each of the pages of search results. However, the use of meta data for the purpose of indexing the search result pages was found to be unreliable. This is because there were many instances of abuse of the meta tags by the webmasters. They did so by using keywords that were irrelevant, in order to increase the number of impressions per page. This was done in order to increase the amount of revenue generated through advertising, although it was not an ethical act on the part of the webmasters.



The most common method to monetise a content-related website at that time was to calculate the cost per one thousand impressions. The webmasters often resorted to providing meta data that was neither accurate nor complete nor consistent in the meta tags. This resulted in the pages giving the rankings for searches that were not relevant and not displaying the rankings for the relevant searches. It was not uncommon for the providers of content on the websites to manipulate the number of attributes within the Hypertext Markup Language (HTML) source of pages either. This was done in order to attempt to rank higher on the pages of search engines.



The heavy reliance upon the factors that were exclusively within the control of the webmasters resulted in the abuse of the early search engines and the manipulation in the ranking of search results. However, they eventually realised the need to display relevant search results on search engines in order to provide better search results to the users. The search engines soon began to adapt in order to make sure that the pages were not filled with content that did not make sense to the users and keywords that were written by webmasters who were unscrupulous in their behaviour.



The search engines resolved the problem by developing a more complex system of ranking the websites with the help of algorithms. They took into consideration the factors that were beyond the control of the webmasters to prevent further manipulation.



Two students of Stanford University, namely Larry Page and Sergey Brin developed a search engine named backrub. It relied upon a complex mathematical algorithm that facilitated the system of rating each web page in the descending order of its prominence. PageRank was the name the duo gave the number calculated by this algorithm. It is not only a function of the quantity, but also that of the strength of inbound links.



The main function of PageRank is to estimate the chances of a given web page reaching the random user of the Web, who surfs and follows the link from one web page to another. Some links are stronger than others, and a web page with a higher PageRank is likely to be reached earlier by a random Web user than one with a lower PageRank.



Page and Brin founded the most popular search engine on the Internet, Google, in the year 1998. Google soon became a hot favourite among Internet users, basically because of its simple design. PageRank and hyperlink analysis were two of the off-page factors that were taken into consideration in addition to the on-page factors. This enables Google to avoid the sort of manipulation that is a common feature of most search engines that only take into account the on-page factors. Webmasters found PageRank too hot to handle, but they immediately set about developing tools that would enable them to build links as well as schemes that would make it easy for them to influence the search engine called Inktomi.

These methods could also be applied in order to gain PageRank. Some websites mainly focused on the exchange, purchase and sale of links, and these activities often took place on a large scale. Some of the link farms or schemes involved the creation of several websites which solely focused on link spamming.



As of the year 2007, search engines have begun to take into account a variety of factors for their ranking algorithms. Many of these factors are undisclosed and their main purpose is to reduce the effect of the various link schemes.



Google, for instance, claims to use more than two hundred different criteria to rank websites. Google, Yahoo and the Microsoft Corporation's Live Search, the three leaders in the search engine industry, keep their page ranking algorithms a closely guarded secret.



The likes of Rand Fishkin, Barry Schwartz, Aaron Wall, Jill Whalen and others, who are big names in the field of search engine optimisation, continuously update their knowledge of the topic by studying different techniques and approaches to achieve the optimisation of search engines. Their findings have been published in blogs and online forums. Practitioners of search engine optimisation also study the patents held by search engines, which helps them understand the algorithms used by them.