The last few weeks I’ve been reading quite a lot on Search Engine Optimisation (SEO) as I’m working on getting some small online products developed in my spare time and on setting up this new blog. In short SEO is the art and science of improving the amount of traffic a website receives as a result of searches through search engines such as Google and Yahoo!. Because of this it’s important to understand at least a little about the way that search engines work to be able to optimise the effectiveness of a website.

Search Engine Operation

Search engines operate by crawling the web using automated programs called ‘bots’ or ’spiders’. Once a page has been crawled it is ‘indexed’, which involves storing it in a large database that the search engine can later use to search; as a side note anyone that has been involved in the development of online search products will appreciate the difficulty and complexity involved in good indexing and searching. When a search is performed the search engine scans the index and retrieves the pages that match the query, which is likely to be a lot so it then has to rank the results.

Ranking

A page is ranked based on…

Relevance :: the level to which the page matches the query or intention submitted to the search engine. The relevance will increase if the term appears multiple times in the document or in parts of the document that are deemed to be ‘important’ such as headers and titles.

Popularity :: this is the part I find interesting. The number of citations and also the quality of the citations is used to assess the popularity of the page and indirectly it’s suitability as a result is in the search. By citations read the number of links into the site from other sites.
Relevance and popularity are then used to give the page a ‘rank’ and determine it’s position in the results. This is a very basic view of the way that this is performed and the top search engines have pretty sophisticated algorithms to perform the ranking and also use other factors to determine the ultimate ordering of the results but this covers the basics.

Optimising the Site

I’m applying this knowledge (and some other things) in a few ways to make my sites perform as well as possible in search results listings…

Site Accessibility :: by following good practices and accessibility guidelines it makes it as easy as possible for bots to read and index the site.

Keywords :: by the intelligent use of the keywords that I am focusing on I should optimise my ability to compete with other sites. For example, if for this blog I focus on the keywords “Software Development” then I’m in for some stiff competition from the likes of IBM. If I focus on a more niche (long-tail) term, “Lean Software” for example, I should stand a much better chance of appearing higher in the results listings.

Links :: by building networks with other sites and gaining links in to my sites I can increase the ranking of my pages. This can be done through social networks or through good use of PR using either dedicated PR sites or respected industry journals and publications online.

A few things to consider here that I found interesting. Links in to your site from other peoples blogs often carry no weight with the popularity ranking system of the search engines as they use the “nofollow” tag which is a message for search engines to not count this link as a ‘vote’. Another interesting point is that it does seem appealing to try and abuse this system by creating dummy sites just to provide links into the real site. This is a bad idea as it could result in being de-listed completely from the search engines. One final interesting point is that the more successful a site becomes in the search results the less of its traffic (as a percentage) is received from the search results themselves. The reason for this is that by this stage there are lots of links in to the site from other locations and these start to become very popular entry points in their own right.

For some further information on the basics of SEO I recommend taking a look at Rand Fishkin’s guide on SEOmoz. I found it to be a really nice overview of the basics and a good platform to getting into the more advanced stuff. It also has links to some nice tools for investigating the number of searches performed on certain keywords.