skip to the main content area of this page

SE SEO SEM History

History of Search Engines and SEM

The Spam Era

1997 – 1999 were the early years of search engine era. SEO mainly consisted of submitting your sites to the search engines. These "voices from the past" are still heard today when this or that SEO company and / or automatic submitting software will claim to do search engine promotion for you by submitting your site to hundreds and thousands of engines and directories. The search engines' indexing programs, called " robots " or " spiders ", looked through all of the HTML code of a page and used some page ranking algorithms that they kept in secret.

Those days were spammers' Heaven: it was rather easy to get your site ranked high. You could just use your keywords lots of times on the page, in the META tags, HTML comments etc. and hide it from the human visitors by making the text tiny or completely invisible with the help of HTML tricks. The search engines didn't have any sophisticated technique to recognize this kind of spam, and such sites usually got high rankings very easily. Today, you still can find some samples of this primitive optimization (however you will have to give it a hard try, because nowadays most of such Web projects have been banned by the search engines for excessive keyword usage).

The only exception was Yahoo which has always been indexed by humans who could in most cases identify and ban spamming pages.

Gradually, search engines started recognizing spam and applying corresponding penalties to Web pages using spam methods. However, search engine optimizers were always one step behind the search engines in finding new ways of cheating the indexing algorithms. Hence each search engine is committed to delivering only relevant results to its visitors, the engines needed to take control away from the spammers and auto-submitters. Many began to try different ways of indexing.

Off-the-page Factors as a Solution

By mid-1999, search portals began using the logic of an ordinary Web surfer to improve the quality of search results. One of the ways independent on page content (and so, on possible spamming effort) was found in clicks tracking. DirectHit (since acquired by AskJeeves / Teoma) introduced a technology that watches which sites are chosen by Web surfers. If the surfers often choose a certain Web site for a given keyword, this site must rise in position for that keyword.

Another way to make indexing algorithms more bulletproof is to rank pages based on how many other pages are linked to it. This "reference" principle has come from regular libraries and archives to actually play the leading role in governing the traffic flow in the Internet nowadays. It has been named " link popularity " and remains a huge factor in ranking.

Both of the above parameters are known as "off-the-page" factors because they are factors that are not directly based on the content on your Web page. The claim of "off-the-page" factors is being liberated from the spammers' influence.

Search Engine Optimization industry has found a temporarily appropriate response by creating the so-called " link farms ". The idea is the following: if the search engines consider how many inbound links a Web site has got, let's create a special link page outside our site which is of low interest to the visitors but which can be fed to the spidering robot. We can place many links to our site on this page and on lots of similar pages, and these links will have their say when the robot comes to index.

The link to a link farmer's Web site was placed on a great number of similar link pages of other participants, and in exchange this farmer had to place their links on his own link page . The official aim of the link farms was to create a community united by total cross-linking, however the hidden goal was achieving high search engine rankings.

While link farms were enjoying great popularity even up to recent time, and loads of special software for link exchange between the link farmers have been written, the search engine spiders became aware of this long ago. The contemporary search engines won't care a straw about such kind of link pages when ranking a site.

Community-Edited Directories

Yahoo was the first and is still one of the most popular search engines. Primarily a human-edited directory, Yahoo faced the problem of finding human resources to deal with an exponentially increasing number of pages queued for indexation. While human-edited directories provide quality results and are almost spam-insensitive, the limited number of editors makes it troublesome to achieve the same perfection in quantity.

Community-edited directories first appeared in 1999. The concept allows for thousands of editors, organized in a system of self-governance, to constantly improve and expand the directory with the new resources. The first to come was the Netscape Open Directory, and the directory (now not functioning) was another early leader., which feeds results into and MSN search, is a newer addition to this category.

The Netscape Open Directory, besides of being community-edited, also was a kind of open source directory. Any developer wanting to create or improve own search portal had access to its content. This resulted in the fact that by the year 2000, listings from Netscape Open Directory started showing up on almost every major search engine.

When the community-edited directories have developed a successful junction of quantity and quality, they have become a weighty part of the search world. Human-edited directories in general tend to play the role of "searcher advocate", since they produce very relevant results for any given search. With the rise in importance of directory listings in 2000, search engine marketers began to concentrate on optimizing their sites for focused, targeted, and quality content.

Paid Listings

In 1999, AltaVista has made an attempt to introduce paid listings, i.e. to take fees for inclusion into the search index. This attempt has turned totally uncompetitive and in addition received a universal denounce. So AltaVista has quitted the project soon. Nevertheless, already by the end of 2000 all major search engines offered some kind of paid listing option. One of the brightest examples is ( previously ), which has began as an independent search player and is now owned by Yahoo. It is still one of the major paid listing / advertising resources, offering ranking for keywords based on an auction system: the higher the bid for a certain keyword, the higher your rank for this keyword. The bid amount is charged every time a user clicks on your site listing, a payment model known as Pay-Per-Click (PPC).

In 2001 most directories and search engines introduced various payment models for paid listings: fees for indexing by the search engine spiders, fees for listings in human-edited directories, ad placement opportunities etc. Yahoo established a one-time submission fee, which was later changed to a yearly submission fee. LookSmart switched from a one-time submission fee to a PPC model. Spider-based engines such as FAST, AltaVista, and Inktomi introduced paid inclusion, i.e. a recurring fee that ensures a site will be listed and regularly re-spidered.

The rise of Google

If someone is asked today about the first search engine to remember, the answer will be Google in 100% cases. Google has started its way to be the King of search engines in 2000 and in 2002 its right for this title has been firmly established, with around 70% of searches done on the Internet. While other search engines were focusing on transforming to universal portals, Google kept a simple and – which has become its distinctive feature – fast interface that solely targeted delivering relevant search results.

Google also developed advanced features such as indexing and searching PDF (portable document format) and SWF (shockwave flash) files. Additionally, Google's sophisticated techniques to use the "off-the-page" factors made it extremely spam-resistant. Google's dominance has become steady in 2000 with Yahoo having switched from Inktomi to Google as secondary search result provider. Now, Yahoo uses the combination of Overture's and own search software and index repository, thus being fully independent on Google, however, without any slightest impact on the dominance of the latter.

Nowadays Google offers numerous tools and services worth speaking about. Here are most important ones we’d like to pay attention to: Google SiteMap, Google Analytics and Page Creator.

Google SiteMap

Google has introduced a new program to speed up the process of crawling large web sites. SiteMap Technology is another way to inform Google about your site and tell its spiders about the main information on the site (the quantity of pages, the frequency of updates, the order you want your pages to be indexed by significance). It also provides statistics and error information about your site. All you need to do is to create an XML sitemap file, put it in the root directory and then submit it ( ). All information about Google SiteMaps can be found here

Google Analytics

Google Analytics ( is a relatively new tool which is aimed at bringing you all kinds of information important for your business: you can determine where your visitors are coming from, which links are the most important in terms of traffic thru them, how much time do the visitors spend on your site’s pages, it tracks your paid links, affiliate campaigns, traces transactions to campaigns and keywords, and provides latency and loyalty metrics. Google Analytics can be used directly from the AdWords interface on condition you have an account. In essence it provides you with a great amount of data in about eighty pre-built reports. To sign up for the service, please, visit

Page Creator

Recently Google launched a new program – Page Creator. It is a web based application and it gives one the opportunity to create and publish web pages ( It allows one to arrange various types of web content and then upload it. Of course it is not for professional web designers or site creators but it is suitable for those who want to create personal sites such as “an online photo tour of your vacation to Bali,” “an overview of the South American precipitation cycle for your science class” as Google remarks. To use the service you don’t have to install any extra software – all you need is a simple setup of Internet Explorer 6.0 or Firefox 1.0 with enabled cookies and JavaScript and a Gmail account (if you don’t have it - Google will host the site for free on its server and the spiders will crawl your site within several hours (though it is stated that no preference will be given to the site). All information about the service is available at

Global Consolidation

By 2001, the results of all major engines were produced from a number of mixed / hybrid sources. Yahoo search results combined Yahoo-directory listings, Overture (PPC) results, and Google results. MSN provided results from Overture (PPC), LookSmart , and Inktomi.

The years 2002 and 2003 brought major reshuffles among search engines: in this period, Google purchased, Yahoo bought Inktomi, and AltaVista and AllTheWeb became a part of Overture. Also, there were many shifts caused by emerging search engine partnerships. Further in this course we will give you a complete and actual chart of relationships between the contemporary search engines.

Here is a table reflecting the major events in the search engine world in 2002-2004:


Google became king of the search engines and the most popular search destination with worldwide Internet users, wrenching market share away from Yahoo.

Feb 2003

Yahoo purchased the Inktomi search index.

Apr 2003

Pay-Per-Click provider Overture bought search engine AltaVista.

Apr 2003

Overture purchased search engine AllTheWeb from FAST Search.

Jun 2003

FAST Search purchased AltaVista Enterprise Search from Overture.

Jun 2003

Microsoft announced their intention to build their own search engine.

Jul 2003

Yahoo purchased Overture (including AltaVista and AllTheWeb).

Oct 2003

LookSmart lost their MSN distribution partnership.

Jan 2004

Pay-Per-Click providers FindWhat and eSpotting merged.

Feb 2004

Yahoo dropped Google results and created their own crawler database.

Mar 2004

AskJeeves purchased Excite.

Apr 2004

Google announced an IPO ( sale of stock to the public ).

Partner Links