Training for Book Authors
|
|
Other Resources for Book Authors & Publishers
|
|
Please Support Our Advertisers
|
|
|
|
|
Today's Google Bots and What They Do
Copyright (c) 2006-2023 Kim Roach, All Rights Reserved
(See This Article in its Original Format.)
Google currently indexes over 8 billion web pages. However, before these pages were placed in the index, they were each crawled by a special spider known as the GoogleBot. Unfortunately, many web masters do not know about the internal workings of this virtual robot.
In fact, Google actually uses a number of spiders to crawl the Web. You can catch these spiders by examining your log files.
This article will attempt to reveal some of the most important Google spiders, their function, and how they affect you as a web master. We'll start with the well-known GoogleBot.
GoogleBot
Googlebot, as you probably know, is the search bot used by Google to scour the web for new pages. Googlebot has two versions, deepbot and freshbot. Deepbot is a deep crawler that tries to folow every link on the web and download as many pages as it can for the Google index. It also examines the internal structure of a site, giving a complete picture for the index.
Freshbot, on the other hand, is a newer bot that crawls the web looking for fresh content. The Google freshbot was implemented to take some of the pressure off of the GoogleBot. The freshbot recalls pages already in the index and then crawls them for new, modified, or updated pages. In this way, Google is better equipped to keep up with the ever-changing web.
This means that the more you update your web site with new, quality content, the more the Googlebot will come by to check you out.
If you'd like to see the Googlebot crawling around your web property more often, you need to obtain quality inbound links. However, there is also one more step that you should take. If you haven't already done so, you should create a Google Sitemap for your site.
Creating a Google sitemap allows you to communicate with Google, telling them about your most important pages, new pages, and updated pages. In return, Google will provide you with some valuable information as well. Google Sitemaps will tell you about
pages it was unable to crawl and links it was unable to follow. This allows you to pinpoint problems and fix them so that you can gain increased exposure in the search results.
The next Google bot in our lineup is known as the MediaBot.
MediaBot - used to analyze Adsense pages
useragent: Mediapartners-Google
MediaBot is the Google crawler for Adsense Publishers. Mediabot is used to determine wich ads Google should display on Adsense pages.
Google recommends that webmasters specifically add a command in their robots.txt file that grants Mediabot access to their entire site. To do this, simply enter the following code into your robots.txt file:
User-agent: Mediapartners-Google*
Disallow:
This will ensure that the MediaBot is able to place relevant Adsense ads on your site.
Keep in mind that ads can still be shown on a page if the MediaBot has not yet visited. If that is the case, the ads chosen will be based on the overall them of the other pages on the site. If no ads can be chosen, the dreaded public service announcements are displayed instead.
There is a strong debate over whether or not the MediaBot is giving websites with Adsense an advantage in the search engines.
Even Matt Cutts has confirmed that the Adsense Mediabot is indexing webpages for Google's main index.
He states, "Pages with AdSense will not be indexed more frequently. It's literally just a crawl cache, so if e.g. our news crawl fetched a page and then Googlebot wanted the same page, we'd retrieve the page from the crawl cache. But there's no boost at all in rankings if you're in AdSense or Google News. You don't get any more pages crawled either."
Matt Cutts claims that your website does not get any advantage by using Adsense. However, in my mind, simply getting your site updated in and of itself is an advantage.
This is very similar to Google Analytics, which also promotes a slightly higher degree of spider activity.
Those who run Google Analystics on their site can expect additional spider activity.
However, you certainly shouldn't depend on any of these tools for getting your site indexed. The key to frequent spidering is having quality inbound links, quality content, and frequent updates.
Have images on your site? If so, you have likely been visited by our next Google spider, the ImageBot.
ImageBot - used to crawl for the Image Search
useragent: GoogleBot-Image
The Imagebot prowls the Web for images to place in their image search. Images are ranked based upon their filename, surrounding text, alt text, and page title.
If you have a website that is primarily image based, then you would definitely want to optimize your images to receive some extra Google traffic.
On the other hand, some web sites may not benefit from Google image search. In most cases, the traffic from the Image search engine is very low quality and rarely converts into buyers. Many people are often just looking for images that they can swipe. So, if you want to save some bandwidth, use your robots.txt file to block ImageBot from accessing your image directory.
One of the few exceptions I would make is if you have a site dedicated to downloadable images.
Our final bot is completely dedicated to the Google Adwords program.
AdsBot - Checks Adwords landing pages for quality
useragent: AdsBot-Google
AdsBot is one of Google's newest spiders. This new crawler is being used to analyze the content of advertising landing pages, which helps determing the Quality score that Google assigns to your ads.
Google uses this Quality score in combination with the amount you are willing to bid to determine the position of your ads. Therefore, ads with a high quality score can rank higher even if other advertisers are paying more than you.
This is one of Google's many efforts to ensure that they are delivering the best results to their users.
Can you still block being spidered? Of course, but it will lower your overall Adwords quality score, which could end up lowering the positioning of your ads. If possible, it is best to give AdsBot complete access to your site.
Today's Google bots are becoming more advanced all the time. However, nothing beats relevant, quality, updated content. Deliver that and the search engines will eat it up.
VOTE ON THIS ARTICLE
Needs Work >>
0 -
1 -
2 -
3 -
4 -
5
<< Excellent Article
Tell our authors what you think about their article.
|
Top-Level Category: Business Online Articles
10 Most Recent Articles Written by Kim Roach
Google Gets Personalized
Written by: Kim Roach |
Distributed: 2006-12-08 |
Word Count: 942 |
Page Views: 3752
| Votes: 13 |
Rating: 1.77
Traditional algorithmic search engines have reached their peak.
Personalized search is a natural and necessary progression for
Google and other search engines as well.
7 Online Marketing Tools for the Smart Webmaster
Written by: Kim Roach |
Distributed: 2006-12-06 |
Word Count: 1144 |
Page Views: 4454
| Votes: 14 |
Rating: 2.36
As an online marketer, it's hard to wade through all of the
misleading information that is running rampant on the Internet.
Who do you trust? Where do you go for the very best information
in online marketing? Well, this article was written to answer
those very questions.
Going Broke on Google Adwords?
Written by: Kim Roach |
Distributed: 2006-12-05 |
Word Count: 1229 |
Page Views: 8142
| Votes: 12 |
Rating: 2.08
You've heard the stories. Click fraud has run rampant on Google,
Yahoo, and MSN. This is evidenced by the numerous law suits that
have been filed.
Creating Your Own Online Radio Station
Written by: Kim Roach |
Distributed: 2006-11-27 |
Word Count: 1138 |
Page Views: 6631
| Votes: 23 |
Rating: 1.91
Podcasting has leveled the playing field for people around the
world. With just a few tools, you can quickly and easily set up
your own talk show on the topic of your choice. All you need is a
computer, a microphone, and a passion to share your knowledge
with others.
Smart Webmasters are Wasting Online Real Estate
Written by: Kim Roach |
Distributed: 2006-11-24 |
Word Count: 1307 |
Page Views: 4080
| Votes: 14 |
Rating: 1.93
Many webmasters are not using their websites' traffic to full
potential. The Web is full of wasted advertising space. Chances
are that even your website has some virtual wasteland. These are
areas that often go unnoticed by even the most experienced
marketers, resulting in the loss of thousands of dollars of
potential revenue.
Hidden Market Research on Ebay, Amazon, and Others
Written by: Kim Roach |
Distributed: 2006-10-12 |
Word Count: 1489 |
Page Views: 8756
| Votes: 28 |
Rating: 1.93
Are you looking to provide interesting content to your visitors?
How about popular in-demand products? Do you know what your
target customer is really searching for online? No worries. You
are about to discover that there are literally thousands of
untapped markets just waiting at your fingertips. To get started,
let's go on a little virtual journey.
Unique Marketing Strategies
Written by: Kim Roach |
Distributed: 2006-10-11 |
Word Count: 1629 |
Page Views: 7402
| Votes: 22 |
Rating: 1.77
As an online marketer I have learned a variety of techniques for
attracting traffic. However, sometimes I stumble upon marketing
strategies that are truly unique. I urge you to read on.
Web Marketing 2.0
Written by: Kim Roach |
Distributed: 2006-10-10 |
Word Count: 1645 |
Page Views: 6744
| Votes: 16 |
Rating: 2.50
If you're like me, then you are probably tired of hearing the
word Web 2.0 passed around the net like a buzz word on steroids.
Google alone currently shows 126,000,000 results for the search
term "web 2.0" at the time of this writing. However, although
this term may be overblown, it does have significant influence on
the current changes taking place on the Internet. Most
importantly, it has many implications for the online marketer.
Franchising Your Online Business
Written by: Kim Roach |
Distributed: 2006-09-29 |
Word Count: 1833 |
Page Views: 4314
| Votes: 22 |
Rating: 2.41
Franchises have proven to be much more successful than the
traditional small business. Almost 80% of new business start-ups
fail each year. Franchising, on the other hand, allows an
entrepreneur to open up a new business based upon a proven brand
and business model.
Essential Programs For Your Virtual Toolbox
Written by: Kim Roach |
Distributed: 2006-09-28 |
Word Count: 947 |
Page Views: 5292
| Votes: 18 |
Rating: 2.28
As an online marketer and webmaster, there are a number of tools
that you will want in your virtual toolbox. To get you started, I
have scoured the net to find some of the best programs to help
webmasters improve their sites, their rankings, and their
productivity. Best of all, each one of these tools is free.
All of Author's Articles on this site: Kim Roach Articles
Most Recent "Business Online" Articles
Four Tips For Creating An Effective Authors' Resource Box
Written by: Bill Platt |
Distributed: 2013-08-27 |
Word Count: 1669 |
Page Views: 8138
| Votes: 18 |
Rating: 2.50
In article marketing, there are many factors that will
affect your articles' ability to promote your website in
the most effective manner. In this article, we will briefly
discuss a couple those factors, before we dive into how to
create an effective Authors' Resource Box, also known as
the About The Author Information.
Website Traffic Generation: Winning the Love of the Traffic Fairy
Written by: Bill Platt |
Distributed: 2013-08-24 |
Word Count: 1149 |
Page Views: 9018
| Votes: 14 |
Rating: 1.86
Some people still believe that all they have to do is build
a website and customers will magically appear on their
site... Once they have built their websites, they wait
anxiously for the "Traffic Fairy" to come and sprinkle its
magic dust on their website too, so that they can make lots
of money like other success website owners.
What Kind Of Backlinks Are Best At Helping Your Website In Google?
Written by: Bill Platt |
Distributed: 2013-08-21 |
Word Count: 439 |
Page Views: 10077
| Votes: 23 |
Rating: 2.74
I was in the Warrior Forum, and someone asked what kinds of
Backlinks are worth pursuing. This is my answer to that
query.
Understanding the Mechanics of Effective Article Marketing and Article Syndication
Written by: Bill Platt |
Distributed: 2013-08-01 |
Word Count: 1399 |
Page Views: 6656
| Votes: 10 |
Rating: 1.80
Visit several marketing blogs or marketing forums, and you
will find dozens of people proclaiming the death of article
marketing as an effective marketing tool. Before we start
agreeing with them, consider this...
Top 10 Reasons Why Online Marketers Fail With Article Marketing
Written by: Bill Platt |
Distributed: 2013-07-31 |
Word Count: 1025 |
Page Views: 7925
| Votes: 13 |
Rating: 2.08
If you've been marketing products online for any length of
time, you've likely heard people talk about the power and
promise of article marketing for the promotion of your
online business. Unfortunately, many online marketers have
tried to utilize article marketing to benefit their online
business and failed. In this article, I document the top ten
reasons why people fail to find success with article
marketing... And how to fix your article marketing
campaigns...
Article Marketing Tip: Stop Trying To Sell From Within Your Articles
Written by: Bill Platt |
Distributed: 2013-07-30 |
Word Count: 1229 |
Page Views: 6104
| Votes: 10 |
Rating: 1.40
The basic premise of article marketing is to present people
information that will be important to them, then at the end
of the article, present information "about the author" and
offer to the reader a means to see how the author can help
the reader further. When done well, article marketing
continues to be one of the most effective methods of
promotion that one could employ online.
The Six Degrees of Internet Marketing
Written by: Bill Platt |
Distributed: 2013-07-30 |
Word Count: 991 |
Page Views: 8529
| Votes: 14 |
Rating: 2.43
The theory behind the Six Degrees of Separation is that all
people on earth can be connected together in as little as
six steps. Now, I know what you might be thinking at this
juncture... That sounds fun and interesting, but what does
that have to do with Internet Marketing? That is a great
question that I am going to answer for you in this
article...
How To Win More Eyeballs for Your Offers and Websites
Written by: Bruce Lansing |
Distributed: 2013-07-29 |
Word Count: 1373 |
Page Views: 6017
| Votes: 11 |
Rating: 1.45
Most people using search engines on the Internet are looking
for answers to questions and solutions to problems. If you
can give people what the answers they want, you will be able
to position your business as one worth hiring today or
tomorrow.
How To Increase Your Rates and Earnings As a Freelance Writer
Written by: Bill Platt |
Distributed: 2013-07-29 |
Word Count: 1191 |
Page Views: 6039
| Votes: 11 |
Rating: 1.55
Freelance writers fill a very important role in today's
business marketplace. They create the on-demand, customized
content that business owners everywhere need to push their
businesses forward. In this article, the author explains
what freelance writers should do to ensure that employers
will pay them well for their services.
Promoting Your Business With Content:: Defining Your Writing Strategy
Written by: Bruce Lansing |
Distributed: 2013-07-25 |
Word Count: 677 |
Page Views: 5743
| Votes: 11 |
Rating: 2.00
Many people suggest that you should write articles on the
basis of the benefits that people will get from reading your
articles. I'd like to suggest that there is a better
reason, and that is to write your articles on the basis of
what problem people will be able to solve, as a result of
reading your articles.
Most Viewed "Business Online" Articles
Attracting Readers to your Book
Written by: Alastair Hall |
Distributed: 2006-05-18 |
Word Count: 708 |
Page Views: 46609
| Votes: 17 |
Rating: 2.06
One of the common and obvious ways to market a self-published
book is having your own website. The harder part is how to
attract your target audience and then convert the sale.
How To Make Visitors Bookmark Your Site
Written by: Cheryl Miller |
Distributed: 2006-10-10 |
Word Count: 536 |
Page Views: 38905
| Votes: 21 |
Rating: 2.19
When visitors bookmark your site it is a sign that you have built
a great website and are pleasing them. Visitors think highly
enough of your site to want to return and possibly tell others
about it. You are not only increasing traffic from your existing
clientele you are encouraging new traffic by means of viral
marketing.
Treat Online 'Guests' With Respect
Written by: Rick Sloboda |
Distributed: 2007-07-27 |
Word Count: 472 |
Page Views: 30403
| Votes: 33 |
Rating: 3.33
Does your website show your customers lack of respect? Review the
following checklist to find out.
How I Got 70,000 Useless Visitors To My Site In One Day! (One Internet Marketer's Analysis of Social Bookmark Traffic)
Written by: Titus Hoskins |
Distributed: 2007-11-06 |
Word Count: 1545 |
Page Views: 28414
| Votes: 10 |
Rating: 1.70
Is social bookmark traffic useless? Is it even worth cultivating
for your site? Can it be used from an online marketing
perspective? Read to discover one Internet Marketer's analysis
of social bookmark/media traffic...
Successful Article Marketers Help Readers Solve Problems
Written by: Bill Platt |
Distributed: 2013-07-24 |
Word Count: 816 |
Page Views: 26141
| Votes: 34 |
Rating: 2.79
Every week, I have the opportunity to speak with people about the
benefits and challenges of using article marketing to promote an
online business. Article marketing is about getting your sales
message in front of potential customers and to get links to your
website. Many interpret this to mean that an article should
directly promote a website within the article, but that approach
is wrong and will reduce one's success using this methodology.
Tips For Building Your First Website
Written by: Benny Tsabba |
Distributed: 2007-05-30 |
Word Count: 1511 |
Page Views: 19469
| Votes: 18 |
Rating: 2.11
New people are coming online everyday. And many of those people
desire to turn their spare time into spare cash. So begins their
journey.
The Importance Of Effective Follow-up
Written by: Kate Smalley |
Distributed: 2007-01-15 |
Word Count: 493 |
Page Views: 14412
| Votes: 22 |
Rating: 2.55
Marketing experts say that following up with clients is crucial
to successful selling because most prospects do not buy the first
time, according to Aweber, a leading autoresponder company.
People must encounter a marketing message multiple times before
making a purchasing decision.
Ten Tips For User Friendly Online Registration Forms
Written by: Jim Romanik |
Distributed: 2009-02-24 |
Word Count: 775 |
Page Views: 12735
| Votes: 15 |
Rating: 0.93
Have you ever registered or attempted to register for an event
and spent half an hour trying to figure out what you needed to
do? Or have you used an online registration form that lost your
data or was so complicated that it seemed like more work than
faxing the form?
Successful Forum JVs and How To Get Them
Written by: Diana Barnum |
Distributed: 2006-06-15 |
Word Count: 648 |
Page Views: 12478
| Votes: 19 |
Rating: 2.11
The forming of common alliances, also referred to as a joint
venture (JV), happens nearly every day in the business world.
One of the most popular means of linking people up online of so
that they can work together on their projects is via forums,
where people of all levels of business and expertise post in
common threads.
Changing Your Company Name: The Good, The Bad and The Unnecessary
Written by: Marcia Yudkin |
Distributed: 2009-07-16 |
Word Count: 520 |
Page Views: 11286
| Votes: 24 |
Rating: 1.79
Wondering whether or not to change your company name? Several
reasons for doing this are legitimate. Other reasons should make
you stop and reconsider.
Highest Ranked "Business Online" Articles
Is Your Business 'Out Of Site'? Employ The Ultimate Marketing Tool
Written by: Rick Sloboda |
Distributed: 2007-01-09 |
Word Count: 445 |
Page Views: 5514
| Votes: 21 |
Rating: 3.76
With the rapid expansion of the digital economy, the web site is
conceivably the most powerful marketing and sales tool you can employ.
Find out what a web site can do for you.
How To Use Your Race To Get Rich
Written by: Herbert Harris |
Distributed: 2009-07-07 |
Word Count: 1407 |
Page Views: 5914
| Votes: 48 |
Rating: 3.71
Amid recent news reports that there continues to be an ongoing
and substantial gap between the net worth of Whites and other
racial groups, it is easy to be distracted and overwhelmed by the
raw data. However, I am immediately reminded of a quote offered
by an economic expert from the past... "There are lies, damned
lies, and statistics!"
Why I'm A 'B-Level Marketer!'
Written by: Willie Crawford |
Distributed: 2007-07-16 |
Word Count: 917 |
Page Views: 6770
| Votes: 26 |
Rating: 3.58
At a recent Internet marketing seminar, I was sitting around the
hotel lobby with approximately 20 other seminar attendees and
speakers. The seminar's host was in attendance, and the
conversation covered a range of internet marketing topics with
extreme frankness. Our proximity to the hotel bar certainly
contributed to our 'conviviality.'
15 fixes for a failing website
Written by: Rick Sloboda |
Distributed: 2007-02-07 |
Word Count: 593 |
Page Views: 3486
| Votes: 25 |
Rating: 3.48
Employ these 15 tactics to improve your search engine rankings,
engage visitors and convert more sales.
The Most Important Ingredient for Info Product Success (is You)
Written by: Judy Murdoch |
Distributed: 2008-04-15 |
Word Count: 981 |
Page Views: 5962
| Votes: 15 |
Rating: 3.47
Last week I was attending a conference and mentioned to someone I
help small business owners create information products. The other
person was very friendly until I said "information products."
Then the temperature in the room dropped about 10-degrees
How to Keep Your Article Content Fresh and Interesting
Written by: Craig Ritsema |
Distributed: 2006-08-29 |
Word Count: 699 |
Page Views: 5499
| Votes: 25 |
Rating: 3.44
Webmasters everywhere are constantly on the lookout for fresh and
interesting content for their website. Along with these content
qualities is the requirement that it be unique. Everyone wants
their website to stand out above the others in the search engine
results. So what is the trick for you to keep a steady supply of
this quality content coming your way?
How To Build A Profitable Twitter Profile
Written by: Harold Hemmings |
Distributed: 2009-08-06 |
Word Count: 1357 |
Page Views: 2657
| Votes: 8 |
Rating: 3.38
Online marketers everywhere are finally starting to tap into
the Twitter Marketplace. Marketers are getting involved in
Twitter at awesome rates, but all seem to have one
unanswered question: How do we actually monetize this
marketplace? It is my goal that by the time you have read
this article, you will have a clear idea of exactly how to
monetize your activities in Twitter.
Increasing Productivity In Your Online Business
Written by: Willie Crawford |
Distributed: 2006-06-14 |
Word Count: 1407 |
Page Views: 7361
| Votes: 40 |
Rating: 3.35
Running an online "empire" of approximately 1600 websites and
blogs, selling a variety of products, services and ideas, I
have to be much more productive than the average person in a
"regular job."
Turn A One-Night-Stand With Google Into A Long-Term Relationship
Written by: Diane Metz |
Distributed: 2007-07-10 |
Word Count: 882 |
Page Views: 2646
| Votes: 6 |
Rating: 3.33
Want to win the favor of Google? Then you've got to build a
long-term relationship with her. Many new SEO strategies are
simply recycled old-tricks. And Google is a smart gal who can see
right through them. Position your website for the future by
understanding what Google really likes â€" and what she
really loathes.
Treat Online 'Guests' With Respect
Written by: Rick Sloboda |
Distributed: 2007-07-27 |
Word Count: 472 |
Page Views: 30403
| Votes: 33 |
Rating: 3.33
Does your website show your customers lack of respect? Review the
following checklist to find out.
|
|
|
|
|
Article Reprint Rights
|

This work is licensed under a
Creative Commons License
You are not required to show the creative commons license notice when you reprint this work.
|
Article Statistics
|
Word Count: 1061
Total Views: 14106
Article Rating: 2.63 of 5
Votes Cast: 16
More Articles By Author:
Articles On This Website
Last Distribution Date:
2006-08-02 09:48:00
Internal ID: #3251
|
|
|