Spiders,
crawlers, and robots, oh my!
Spiders, crawlers, and robots are the hungry little programs
that gobble up and process information on the web for the
search engines. They are the gatherers. They are the mechanisms
that make finding new information and updating existing information
possible by rapidly traveling through your site and all other
sites on the web and caching your content in their database
to be indexed and to turn up in their search engine’s
results. This is important to understand because if you do
not understand how these little intangible “bots”
work, then the chances of your site being included on a search
engine’s index for any period of time are pretty slim.
There are once again only two ways one of these “bots”
can find your web pages to index them. The first is they simply
find your pages because you tell them where they are by submitting
the pages to the engine (the direct entry), or secondly, the
engines find your pages on their own through a link from another
website (the referral). If you have no outside links from
other sites on the web, it is quite possible that your site
can be dropped from an engine’s index because it never
found a way in its rounds, so whichever method you choose
to use to get indexed, make sure you have links from a referring
site set up for the future. An isolated web site (a site with
no inbound links pointing to it) is not good. Some SEO’s
even believe that if the engines find your site indirectly
without submitting your pages, this actually helps your pages
position better. These spiders actually prefer to find pages
on their own without having to be told where to look. In other
words, if I personally had to choose between submitting my
pages to the engines directly or getting a couple of popular
websites with similar themes to my own to place links on their
site to mine, I would most definitely do the second.
So do yourself a favor and find some websites like this to
befriend and exchange links with them. Put a link to their
site on yours to return the favor. It is extremely important
to have outside links pointing to your site (“link popularity”
is the term used to describe this). After all, a spider can’t
index your site if it can’t find a way into it!
|
