Themes-based spiders represent search
engine companies' latest attempt to deliver
high-quality results to searchers. The
change is happening relatively slowly,
which gives webmasters a chance to measure
the impact on their ranking and modify
their sites accordingly. Find out if your
site is ready for a visit from a
A New Way To Rank Web Sites
At its most basic level, a theme is a
simple description of your site's focus.
If you've already optimized your site using
TITLE Tag and targeted
Keywords, you should have a pretty good
idea of your site's theme.
But do the search
engine spiders view it the same way? A
page-based spider will look at and index
each individual page of a Web site. You
could conceivably have hundreds of pages
listed individually in a search engine's
database, all of them optimized for
different keywords. Themes-based spiders
also look at individual pages in a site,
but then combine the results and analyze
the site in its entirety.
For example, imagine a site that contains
several pages devoted to drawn-thread
embroidery patterns and other pages that
discuss endangered desert tortoises and
Nevada ranchers. A page-based spider could
conceivably give both topics a high search
results ranking if the individual pages
were optimized correctly.
Not so with a themes-based spider. When it
mixes the keywords and content from the
different topics, it might decide that the
entire site is really devoted to Nevada
ranchers who use drawn-thread techniques to
embroider tortoises. That keyword phrase
might indeed get the site a #1 search
results rank, but chances are it would
generate very little traffic and isn't at
all what the webmaster had in mind.
How A Themes-Based Spider Works
Currently, there is no "pure"
themes-based spider indexing sites. Some
search engines have added themes to their
overall ranking algorithm, but the results
are integrated into the more traditional
ranking criteria. Those engines include:
Google, Inktomi, AltaVista, Excite, Lycos,
Fast, and WebCrawler.
Just like a page-based spider, a
themes-based spider crawls through your
site cataloging the following information:
- Your top-level
domain name and
- Individual pages' TITLE
- META Tags data.
- HTML Tags inside the
document's BODY section (Header
Tags, hyperlinks, etc.).
- Text content of the page.
- Inbound and outbound links.
So far, this isn't any different from a
page-based spider, but the themes spider
isn't finished yet. When the spider
finishes crawling through the site, it then
determines the site's theme by evaluating
the pages as a whole, not individually.
Steps in this process include:
- Match keywords from the
META tag with page text to
identify keywords used most
frequently in the site.
- Identify and match keyword
phrases in the site.
- Weight the keywords,
keyword phrases, and links
using traditional ranking
methods that attach more
importance to keywords
contained in TITLE and header
- Give the site one entry in
the database according to the
theme the spider has selected.
If you can't accurately describe your site
in one sentence and then optimize the page
content to reflect that focus, then you
have a problem. Your site's traffic will
suffer if the spider assigns a theme that
doesn't match your intended focus or
Optimizing Your Site For Themes
There's no reason to panic and tear apart a
site that is doing well. But if you're
having trouble getting to the top in some
engines, the lack of a coherent theme may
well be part of the problem.
Think about it: the absence of tightly
focused content and keywords will hurt you
in any search engine, whether it's
page-based or themes-based. Themes-based
spiders just magnify the problem.
Optimizing your site for themes is not much
different than more familiar search
engine optimization techniques.
As themes-based spiders grow in importance
though, you may find it more difficult to
optimize individual pages on your site for
widely differing topics. Keep themes-based
spidering in mind as you create new sites
and redesign existing ones. Consider
breaking up a single, multi-focused site
into different domains and linking between
them instead of trying to combine many
varied topics into a single site.
Remember The Basics
The basics of good site promotion haven't
changed: a TITLE
tags, and good content with targeted
keywords and their synonyms sprinkled
liberally throughout. Pay close attention
to your inbound and outbound links: a high
popularity score increases your rank
with both traditional spiders and
You spend so much valuable time designing a
great site. Make sure that
everyone can find it!