Is Your Site Spider Friendly?

What kind of attention span does a search engine spider have? It depends. Some have the patience to dig deep into your site and promise to follow every link. Others may get bored before they finish indexing a whole page. Carefully organize your page content and Web site to ensure that spiders index the whole thing - not just bits and pieces.

Make Your Points Early And Well

Human visitors want to see your most important content at the top of the page, highlighted if necessary with color and header tags. But realize that spiders see the page differently than human visitors.

Search engine spiders look at your content and assume that:

  • Earlier content is more important than what's farther down on the page.

  • Text contained in header tags is more important than other text.

  • Links and header tags that contain keywords matching the META tag keywords are more important than those that don't.

These used to be a very important component of all search engine algorithms, but lately has been overshadowed by other ranking criteria like link popularity and themes. Don't ignore them though: they still affect your rank on many engines and help boost you above competing sites.

Put Your Content On Top

Even if visitors see your content on top of the page in their browsers, spiders might not find it if your HTML code is poorly written. If so, your best keywords and keyword phrases may be the last thing the spider reads.

Some design techniques that work great for human visitors leave spiders out in the cold. Make them more spider-friendly like this:

  • External JavaScript files: Complex JavaScript code inserted in the document's HEAD tag takes up a huge amount of space. Use an external JavaScript file to reduce page size or place your JavaScript code at the bottom of the page whenever possible.

  • Cascading Style Sheets: Use style sheets to reduce bloated code and decrease download time. Style sheets eliminate extraneous HTML code, so the spider can get straight to your content.

  • Tables: Keep your table structure as simple as possible. Some spiders tend to get lost in a web of nested tables where the content is buried deep inside a complex table structure.

Be particularly careful if you're using a WYSIWYG editor:

  • Bloated HTML code: This is a problem with many WYSIWYG editors. It's easy to forget what's happening to your HTML code when you're cutting and pasting text and images without writing any actual code yourself.

    Microsoft Office users often don't think they need an HTML editor at all: they can select the Save as Web option on their File menu and convert it into an HTML document. But all those files are big (really big!) and almost impossible to hand edit later.

  • Editor-generated JavaScript: Many editors let you include basic JavaScript effects like drop-down menus and rollovers automatically. While easy, this method also writes big code. A simple rollover event that takes a few lines to hand code might be 10 times that size if it's editor-generated.

Once all visitors (human and spider) see your content up front, it's time to consider the overall structure of your site, not just individual page layout and coding.

Flatter Is Better

Try to design your site with as few levels as possible. Too many sites have this sort of structure:

Level 1: Splash page with a "click here to enter" link.
Level 2: Home page with little content other than navigation links.
Level 3: Products page whose only content is a list of product categories. Visitors click on the category that interests them.
Level 4: Specific product categories with keyword-rich descriptions of each product.

Visitors have to wait for 3 or more pages to load before ever finding the information they want, while search engine spiders search through multiple pages before finding good content to index.

A splash page adds an extra level to your Web site, pushing all of your other content down one level. This increases the chance that both humans and spiders will miss it.

Keep your home page short, but make it content-rich. Reference major topics and include important keywords, then link them to keyword-rich 2nd and 3rd-level pages for more information. A simple "Products" link in your navigation menu doesn't help either human or spider visitors. Also include a "Products" paragraph that briefly lists the type of items you sell:

"We offer the Web's largest selection of 
  premium cat food, dog food, dog 
    grooming supplies, and pet toys!"

Make sure that your keyword phrases "premium cat food, pet toys, etc." all link to the internal site pages that discuss them more completely. This gives spiders a direct link to your important content and it can help your overall page rank because some ranking algorithms place more weight on keywords contained in links.

Deep Submit Your Pages

But do you really need to worry about how many site levels you have? Most of the major search engines now boast that they do a "deep crawl" on all Web sites they visit and therefore index the entire site.

That's not always the case however. Most crawl down a few levels deep into your site and then they're off to the next site.

Search engines claim to index hundreds of millions, even billions of pages. With numbers that size, it's likely that they're going miss a few important pages here and there.

Probably when you started promoting your Web site, you submitted your home page to the top search engines and waited for the spiders to visit and crawl your site.

That may not be good enough. If you use a technique called deep submit, you don't just hope the spiders find your internal pages; you actually submit those too!

This is where an automated search engine submission tool becomes helpfull. Use it to submit any page in your site to your choice of 100 search engines.

Don't risk being overlooked. Design a site that appeals to both humans and search engine spiders, then use deep submit to achieve good search engine exposure for your whole site, not just a page or two.

Rate This Tip:
Not Useful Useful Very Useful   

Copyright by Steve Nearman (The Master's Touch - 2003- all rights
reserved. No part of this web site can be reproduced in any form without written permission.