Search Engine Visibility and Site Crawlability

By Eric Enge, Search Engine Watch

At the recent SES Chicago. Laura Thieme and Matt Bailey, both experienced presenters, showed how webmasters could tame the dynamic Web site beast and improve search engine visibility and site crawlability.

This is the first of two articles outlining the SEO problems they identified with dynamic Web sites and solutions. I'll expand on both.

At the very highest level, dealing with large dynamic Web sites requires IT and marketing to collaborate. The person responsible for the technical search engine optimization is an important member of the team. Without the cooperation of technical SEO support staff, your Web site can end up poorly optimized for search engines.

Keyword Research and Deployment

Do your keyword research up front. You need to understand the language potential customers use to find the products or services you offer. You also want to understand related topics that interest these potential customers.

You then want to match up this understanding of the marketplace with the available content you have, or are willing to develop, and design a logical and clean hierarchy around that.

Laura said embedding keywords into your page names (i.e. the URI) is a powerful SEO tactic. In fact, she recommends existing sites embed the most important keywords for each page into their URI (and then 301 redirecting the old page location to the new page location).

Don't use the same titles and headers for all the pages of your site. That's another reason to do keyword research. The long tail of keywords enable you to create a different title and header for every page of the site.

Large sites also have issues with pages being found and indexed. Make sure the most important keywords for your site appear in the title and header for the home page.

Robots.txt Files

Matt pointed out that it's surprising how many sites have problems with their robots.txt file. He provided a tutorial on how robot.txt files work. You can also find good tutorials on the Web.

I've seen many sites that have problems with Robots.txt. While it's a powerful tool for directing the way search engine crawlers see your site, it's easy to make a mistake. A single mistake can have catastrophic consequences. So use robots.txt, just use the file with great care.

To illustrate how easy it is to make a mistake, we worked with a client a while back that implemented new versions of their site on a staging server separate from the one used for the main site. This allowed them to look at the site live before launching it.

They didn't want this duplicate site indexed by the search engines, so they implemented a robots.txt as follows:

User-agent: *
Disallow: /

Experienced readers are already cringing because they know where this is going. Unfortunately, one day during a site update they copied the whole site from the staging server to the live server, including the robots.txt file.

It took 3 weeks before anyone noticed, and that happened because traffic to the site was already crashing. Ouch. Again, use it, but be careful.

Site Maps

The Sitemap is an XML file that lists URLs for a site, as well as when each URL was last updated, how often it usually changes, and its importance relative to other URLs in the site. The protocol was designed to enable search engines crawl the site more intelligently. In principle, this should help the search engine find all of the pages on your site more easily than crawling it naturally.

Matt had a great point when he told the audience you're better off letting your site being found naturally by the crawler. He referred to Sitemaps as a "fast track to the supplemental index".

I couldn't agree more. We don't use Sitemaps on our clients' sites.

The two best solutions for the crawlability problem -- lots of pages not being indexed -- the following two steps are the best solution:

1. Build an efficient and clean global navigation system on your site. There is no better way to help a crawler find your site than a logical and simple navigation structure. If you do this, the crawler will find its way around.
2. Get third party sites to link to your site. No on-site search engine optimization strategy will work unless you get these links.