|
Article Directory
|
+ Authors
+ Articles
|
|
|
|
Today's Google Bots and What They Do
Copyright (c) 2006-2012 Kim Roach, All Rights Reserved
(See This Article in its Original Format.)
Google currently indexes over 8 billion web pages. However, before these pages were placed in the index, they were each crawled by a special spider known as the GoogleBot. Unfortunately, many web masters do not know about the internal workings of this virtual robot.
In fact, Google actually uses a number of spiders to crawl the Web. You can catch these spiders by examining your log files.
This article will attempt to reveal some of the most important Google spiders, their function, and how they affect you as a web master. We'll start with the well-known GoogleBot.
GoogleBot
Googlebot, as you probably know, is the search bot used by Google to scour the web for new pages. Googlebot has two versions, deepbot and freshbot. Deepbot is a deep crawler that tries to folow every link on the web and download as many pages as it can for the Google index. It also examines the internal structure of a site, giving a complete picture for the index.
Freshbot, on the other hand, is a newer bot that crawls the web looking for fresh content. The Google freshbot was implemented to take some of the pressure off of the GoogleBot. The freshbot recalls pages already in the index and then crawls them for new, modified, or updated pages. In this way, Google is better equipped to keep up with the ever-changing web.
This means that the more you update your web site with new, quality content, the more the Googlebot will come by to check you out.
If you'd like to see the Googlebot crawling around your web property more often, you need to obtain quality inbound links. However, there is also one more step that you should take. If you haven't already done so, you should create a Google Sitemap for your site.
Creating a Google sitemap allows you to communicate with Google, telling them about your most important pages, new pages, and updated pages. In return, Google will provide you with some valuable information as well. Google Sitemaps will tell you about
pages it was unable to crawl and links it was unable to follow. This allows you to pinpoint problems and fix them so that you can gain increased exposure in the search results.
The next Google bot in our lineup is known as the MediaBot.
MediaBot - used to analyze Adsense pages
useragent: Mediapartners-Google
MediaBot is the Google crawler for Adsense Publishers. Mediabot is used to determine wich ads Google should display on Adsense pages.
Google recommends that webmasters specifically add a command in their robots.txt file that grants Mediabot access to their entire site. To do this, simply enter the following code into your robots.txt file:
User-agent: Mediapartners-Google*
Disallow:
This will ensure that the MediaBot is able to place relevant Adsense ads on your site.
Keep in mind that ads can still be shown on a page if the MediaBot has not yet visited. If that is the case, the ads chosen will be based on the overall them of the other pages on the site. If no ads can be chosen, the dreaded public service announcements are displayed instead.
There is a strong debate over whether or not the MediaBot is giving websites with Adsense an advantage in the search engines.
Even Matt Cutts has confirmed that the Adsense Mediabot is indexing webpages for Google's main index.
He states, "Pages with AdSense will not be indexed more frequently. It's literally just a crawl cache, so if e.g. our news crawl fetched a page and then Googlebot wanted the same page, we'd retrieve the page from the crawl cache. But there's no boost at all in rankings if you're in AdSense or Google News. You don't get any more pages crawled either."
Matt Cutts claims that your website does not get any advantage by using Adsense. However, in my mind, simply getting your site updated in and of itself is an advantage.
This is very similar to Google Analytics, which also promotes a slightly higher degree of spider activity.
Those who run Google Analystics on their site can expect additional spider activity.
However, you certainly shouldn't depend on any of these tools for getting your site indexed. The key to frequent spidering is having quality inbound links, quality content, and frequent updates.
Have images on your site? If so, you have likely been visited by our next Google spider, the ImageBot.
ImageBot - used to crawl for the Image Search
useragent: GoogleBot-Image
The Imagebot prowls the Web for images to place in their image search. Images are ranked based upon their filename, surrounding text, alt text, and page title.
If you have a website that is primarily image based, then you would definitely want to optimize your images to receive some extra Google traffic.
On the other hand, some web sites may not benefit from Google image search. In most cases, the traffic from the Image search engine is very low quality and rarely converts into buyers. Many people are often just looking for images that they can swipe. So, if you want to save some bandwidth, use your robots.txt file to block ImageBot from accessing your image directory.
One of the few exceptions I would make is if you have a site dedicated to downloadable images.
Our final bot is completely dedicated to the Google Adwords program.
AdsBot - Checks Adwords landing pages for quality
useragent: AdsBot-Google
AdsBot is one of Google's newest spiders. This new crawler is being used to analyze the content of advertising landing pages, which helps determing the Quality score that Google assigns to your ads.
Google uses this Quality score in combination with the amount you are willing to bid to determine the position of your ads. Therefore, ads with a high quality score can rank higher even if other advertisers are paying more than you.
This is one of Google's many efforts to ensure that they are delivering the best results to their users.
Can you still block being spidered? Of course, but it will lower your overall Adwords quality score, which could end up lowering the positioning of your ads. If possible, it is best to give AdsBot complete access to your site.
Today's Google bots are becoming more advanced all the time. However, nothing beats relevant, quality, updated content. Deliver that and the search engines will eat it up.
Follow "The Phantom Writers" on Twitter
(@phantomwriters)
to be notified when new articles are made available.
VOTE ON THIS ARTICLE
Needs Work >>
0 -
1 -
2 -
3 -
4 -
5
<< Excellent Article
Tell our authors what you think about their article.
|
Top-Level Category: Business Online Articles
10 Most Recent Articles Written by Kim Roach
Google Gets Personalized
Written by: Kim Roach |
Distributed: 2006-12-08 |
Word Count: 942 |
Page Views: 1253
| Votes: 5 |
Rating: 1.40
Traditional algorithmic search engines have reached their peak.
Personalized search is a natural and necessary progression for
Google and other search engines as well.
7 Online Marketing Tools for the Smart Webmaster
Written by: Kim Roach |
Distributed: 2006-12-06 |
Word Count: 1144 |
Page Views: 1907
| Votes: 6 |
Rating: 3.00
As an online marketer, it's hard to wade through all of the
misleading information that is running rampant on the Internet.
Who do you trust? Where do you go for the very best information
in online marketing? Well, this article was written to answer
those very questions.
Going Broke on Google Adwords?
Written by: Kim Roach |
Distributed: 2006-12-05 |
Word Count: 1229 |
Page Views: 2071
| Votes: 4 |
Rating: 2.25
You've heard the stories. Click fraud has run rampant on Google,
Yahoo, and MSN. This is evidenced by the numerous law suits that
have been filed.
Creating Your Own Online Radio Station
Written by: Kim Roach |
Distributed: 2006-11-27 |
Word Count: 1138 |
Page Views: 2043
| Votes: 12 |
Rating: 2.25
Podcasting has leveled the playing field for people around the
world. With just a few tools, you can quickly and easily set up
your own talk show on the topic of your choice. All you need is a
computer, a microphone, and a passion to share your knowledge
with others.
Smart Webmasters are Wasting Online Real Estate
Written by: Kim Roach |
Distributed: 2006-11-24 |
Word Count: 1307 |
Page Views: 1263
| Votes: 6 |
Rating: 2.00
Many webmasters are not using their websites' traffic to full
potential. The Web is full of wasted advertising space. Chances
are that even your website has some virtual wasteland. These are
areas that often go unnoticed by even the most experienced
marketers, resulting in the loss of thousands of dollars of
potential revenue.
Hidden Market Research on Ebay, Amazon, and Others
Written by: Kim Roach |
Distributed: 2006-10-12 |
Word Count: 1489 |
Page Views: 1547
| Votes: 17 |
Rating: 2.00
Are you looking to provide interesting content to your visitors?
How about popular in-demand products? Do you know what your
target customer is really searching for online? No worries. You
are about to discover that there are literally thousands of
untapped markets just waiting at your fingertips. To get started,
let's go on a little virtual journey.
Unique Marketing Strategies
Written by: Kim Roach |
Distributed: 2006-10-11 |
Word Count: 1629 |
Page Views: 2617
| Votes: 14 |
Rating: 1.71
As an online marketer I have learned a variety of techniques for
attracting traffic. However, sometimes I stumble upon marketing
strategies that are truly unique. I urge you to read on.
Web Marketing 2.0
Written by: Kim Roach |
Distributed: 2006-10-10 |
Word Count: 1645 |
Page Views: 2153
| Votes: 8 |
Rating: 3.00
If you're like me, then you are probably tired of hearing the
word Web 2.0 passed around the net like a buzz word on steroids.
Google alone currently shows 126,000,000 results for the search
term "web 2.0" at the time of this writing. However, although
this term may be overblown, it does have significant influence on
the current changes taking place on the Internet. Most
importantly, it has many implications for the online marketer.
Franchising Your Online Business
Written by: Kim Roach |
Distributed: 2006-09-29 |
Word Count: 1833 |
Page Views: 1598
| Votes: 15 |
Rating: 2.60
Franchises have proven to be much more successful than the
traditional small business. Almost 80% of new business start-ups
fail each year. Franchising, on the other hand, allows an
entrepreneur to open up a new business based upon a proven brand
and business model.
Essential Programs For Your Virtual Toolbox
Written by: Kim Roach |
Distributed: 2006-09-28 |
Word Count: 947 |
Page Views: 1583
| Votes: 10 |
Rating: 2.60
As an online marketer and webmaster, there are a number of tools
that you will want in your virtual toolbox. To get you started, I
have scoured the net to find some of the best programs to help
webmasters improve their sites, their rankings, and their
productivity. Best of all, each one of these tools is free.
All of Author's Articles on this site: Kim Roach Articles
Most Recent "Business Online" Articles
Sell Yourself, Not Your Products or Services
Written by: Bill Platt |
Distributed: 2012-01-02 |
Word Count: 989 |
Page Views: 2755
| Votes: 5 |
Rating: 4.00
Sell yourself, not your products or services. The point we
need to make is that your products and services will change
over time as your business adapts to the realities of the
marketplace.
Work From Home - Make Money Online With EBay Store
Written by: David Miller |
Distributed: 2011-12-12 |
Word Count: 2105 |
Page Views: 354
| Votes: 1 |
Rating: 2.00
Find out how to present your items, as well as other key factors
to profit from your eBay Store.
Bring Your Photography To Life With Stunning Video
Written by: Adam Harding |
Distributed: 2011-12-12 |
Word Count: 684 |
Page Views: 459
I want to let you in on my little secret. It is about how I bring
photographs to life by creating professional video slideshows
that look like they cost hundreds of dollars.
Leveraging the Social in Social Media
Written by: Enzo F. Cesario |
Distributed: 2011-09-21 |
Word Count: 907 |
Page Views: 395
| Votes: 1 |
Rating: 1.00
Social media marketing is about the conversation, not the
pitch. Having conversations with people is nice, but the
goal of a brand is ultimately to get people interested in
spending their time or money on the brand, after all. What
does having this conversation accomplish, and just how does
a brand use that conversation to get the critical conversion
from conversation to customers?
Blog Writing - Finding the Right Person
Written by: Enzo F. Cesario |
Distributed: 2011-09-21 |
Word Count: 884 |
Page Views: 630
There are a host of articles and sites covering advice on
how to address various topics, but one area that doesn't
get a lot of attention is what "person" to write the blog
in. Each linguistic person has its own strengths and
weaknesses, depending on the message that needs to be
covered. Ultimately, most will simply write in whatever
person is most natural to them, but there is some merit in
taking a bit of time to think about the implications,
advantages and disadvantages of each.
That First Impression: Title Considerations
Written by: Enzo F. Cesario |
Distributed: 2011-09-19 |
Word Count: 860 |
Page Views: 355
While it certainly isn't as much work as crafting the body
of a superior article, selecting a title for a piece is
definitely an important element of the entire process of
content creation. Here are some thoughts on crafting a
perfect title.
5 Tips For Writing Great Content For Your Blog
Written by: Lee Dobbins |
Distributed: 2011-09-16 |
Word Count: 987 |
Page Views: 405
If you want people to come back to your blog over and over
again, then you need to provide great web content. This may
seem like an impossible task if you aren’t a natural born
writer. Here are 5 tips that will help even the least
experienced writer create blog content that will entice even
first time visitors to become frequent readers.
Understanding Cloud Computing for SMBs
Written by: Audre Hill |
Distributed: 2011-09-16 |
Word Count: 449 |
Page Views: 513
| Votes: 2 |
Rating: 5.00
The buzz about cloud migration, or taking your business to the
cloud, is all over the internet. But the geek speak and sales
hype make understanding the cloud difficult for small and
mid-sized business owners. Read this to clear up your cloud
confusion.
Dynamic Content: Using Analytics to Write Your Blog
Written by: Enzo F. Cesario |
Distributed: 2011-09-16 |
Word Count: 873 |
Page Views: 425
The role of web analytics in setting useful metrics for a
site has been discussed in-depth in any number of places,
but this is far from the only useful function that web
analytic practices can offer. As in any field, a little
thinking outside the box can offer extensive rewards to the
creative researcher. Perhaps most usefully, creative
cultivation of web analytics can help conquer blogger's
block and help a good writer come up with some unique angles
for content they thought they had peaked out on.
Social Media Marketing Tools – Putting Your Finger on the Pulse of the Web
Written by: Enzo F. Cesario |
Distributed: 2011-09-14 |
Word Count: 856 |
Page Views: 343
Social media marketing is the home of successful brand
promotion, and finding the right mix of tools to take
advantage of it has become the defining issue of the modern
brand. Knowing the focus of each of these tools and the best
way to use them is the key to bringing all this power under
control and using it to help promote a brand to its full
potential.
Most Viewed "Business Online" Articles
Attracting Readers to your Book
Written by: Alastair Hall |
Distributed: 2006-05-18 |
Word Count: 708 |
Page Views: 42520
| Votes: 9 |
Rating: 2.11
One of the common and obvious ways to market a self-published
book is having your own website. The harder part is how to
attract your target audience and then convert the sale.
How To Make Visitors Bookmark Your Site
Written by: Cheryl Miller |
Distributed: 2006-10-10 |
Word Count: 536 |
Page Views: 34815
| Votes: 13 |
Rating: 2.38
When visitors bookmark your site it is a sign that you have built
a great website and are pleasing them. Visitors think highly
enough of your site to want to return and possibly tell others
about it. You are not only increasing traffic from your existing
clientele you are encouraging new traffic by means of viral
marketing.
Treat Online 'Guests' With Respect
Written by: Rick Sloboda |
Distributed: 2007-07-27 |
Word Count: 472 |
Page Views: 27199
| Votes: 25 |
Rating: 3.76
Does your website show your customers lack of respect? Review the
following checklist to find out.
How I Got 70,000 Useless Visitors To My Site In One Day! (One Internet Marketer's Analysis of Social Bookmark Traffic)
Written by: Titus Hoskins |
Distributed: 2007-11-06 |
Word Count: 1545 |
Page Views: 23763
| Votes: 8 |
Rating: 2.13
Is social bookmark traffic useless? Is it even worth cultivating
for your site? Can it be used from an online marketing
perspective? Read to discover one Internet Marketer's analysis
of social bookmark/media traffic...
Successful Article Marketers Help Readers Solve Problems
Written by: Bill Platt |
Distributed: 2008-10-29 |
Word Count: 816 |
Page Views: 18814
| Votes: 23 |
Rating: 3.26
Every week, I have the opportunity to speak with people about the
benefits and challenges of using article marketing to promote an
online business. Article marketing is about getting your sales
message in front of potential customers and to get links to your
website. Many interpret this to mean that an article should
directly promote a website within the article, but that approach
is wrong and will reduce one's success using this methodology.
Tips For Building Your First Website
Written by: Benny Tsabba |
Distributed: 2007-05-30 |
Word Count: 1511 |
Page Views: 14636
| Votes: 9 |
Rating: 2.00
New people are coming online everyday. And many of those people
desire to turn their spare time into spare cash. So begins their
journey.
Successful Forum JVs and How To Get Them
Written by: Diana Barnum |
Distributed: 2006-06-15 |
Word Count: 648 |
Page Views: 6635
| Votes: 11 |
Rating: 2.18
The forming of common alliances, also referred to as a joint
venture (JV), happens nearly every day in the business world.
One of the most popular means of linking people up online of so
that they can work together on their projects is via forums,
where people of all levels of business and expertise post in
common threads.
Increasing Productivity In Your Online Business
Written by: Willie Crawford |
Distributed: 2006-06-14 |
Word Count: 1407 |
Page Views: 4327
| Votes: 32 |
Rating: 3.72
Running an online "empire" of approximately 1600 websites and
blogs, selling a variety of products, services and ideas, I
have to be much more productive than the average person in a
"regular job."
Who Else Wants to Know How-To Write an Effective Advertisement?
Written by: Bill Platt |
Distributed: 2011-08-25 |
Word Count: 1431 |
Page Views: 4296
| Votes: 22 |
Rating: 2.59
Webmasters struggle everyday with the question of how they should
advertise their website. There are so many advertising venues; it
is hard to know which is the right venue or the right combination
of advertising venues.
A Recent Issue Of Playboy Revealed...
Written by: Bill Platt |
Distributed: 2010-01-19 |
Word Count: 1658 |
Page Views: 4217
| Votes: 1 |
Rating: 4.00
If you have ever spent money trying to use article
marketing, and you failed to generate any measurable success
with your articles, then this article will show you why you
most likely failed to see your articles generate a solid
return-on-investment for your online business...
Highest Ranked "Business Online" Articles
What Kind Of Backlinks Are Best At Helping Your Website In Google?
Written by: Bill Platt |
Distributed: 2009-11-09 |
Word Count: 439 |
Page Views: 1820
| Votes: 5 |
Rating: 4.80
I was in the Warrior Forum, and someone asked what kinds of
Backlinks are worth pursuing. This is my answer to that
query.
How to Keep Your Article Content Fresh and Interesting
Written by: Craig Ritsema |
Distributed: 2006-08-29 |
Word Count: 699 |
Page Views: 2156
| Votes: 17 |
Rating: 4.18
Webmasters everywhere are constantly on the lookout for fresh and
interesting content for their website. Along with these content
qualities is the requirement that it be unique. Everyone wants
their website to stand out above the others in the search engine
results. So what is the trick for you to keep a steady supply of
this quality content coming your way?
Is Your Business 'Out Of Site'? Employ The Ultimate Marketing Tool
Written by: Rick Sloboda |
Distributed: 2007-01-09 |
Word Count: 445 |
Page Views: 2200
| Votes: 19 |
Rating: 4.16
With the rapid expansion of the digital economy, the web site is
conceivably the most powerful marketing and sales tool you can employ.
Find out what a web site can do for you.
How To Use Your Race To Get Rich
Written by: Herbert Harris |
Distributed: 2009-07-07 |
Word Count: 1407 |
Page Views: 2573
| Votes: 40 |
Rating: 4.05
Amid recent news reports that there continues to be an ongoing
and substantial gap between the net worth of Whites and other
racial groups, it is easy to be distracted and overwhelmed by the
raw data. However, I am immediately reminded of a quote offered
by an economic expert from the past... "There are lies, damned
lies, and statistics!"
Secrets of Domain Parking - An Innovative and Easier Way to Make Money
Written by: John Khu |
Distributed: 2009-02-27 |
Word Count: 484 |
Page Views: 1392
| Votes: 7 |
Rating: 4.00
Using a good domain parking services to monetize expired domain
names seems to be a neglected aspect with a number of domain
names traders. Domain parking is also an unfamiliar concept to
many inexperienced domain name traders. Read this article to
better understand domain parking and how it can be of benefit to
you.
Expired Domain Gains - Simple Ideas to Create an Income Base
Written by: John Khu |
Distributed: 2009-07-31 |
Word Count: 549 |
Page Views: 3232
| Votes: 14 |
Rating: 4.00
You can use a number of simple methods and plans to create a
solid income base that earns a monthly ongoing income base.
For many expired domain entrepreneurs (both full-time and
part time), expired domains offer an opportunity to
supplement their current income with money earned out of
dealing with expired domains.
Sell Yourself, Not Your Products or Services
Written by: Bill Platt |
Distributed: 2012-01-02 |
Word Count: 989 |
Page Views: 2755
| Votes: 5 |
Rating: 4.00
Sell yourself, not your products or services. The point we
need to make is that your products and services will change
over time as your business adapts to the realities of the
marketplace.
The Most Important Ingredient for Info Product Success (is You)
Written by: Judy Murdoch |
Distributed: 2008-04-15 |
Word Count: 981 |
Page Views: 1429
| Votes: 13 |
Rating: 3.92
Last week I was attending a conference and mentioned to someone I
help small business owners create information products. The other
person was very friendly until I said "information products."
Then the temperature in the room dropped about 10-degrees
How To Build A Website That Will Rank Well In The Search Engines?
Written by: Fred Schebesta |
Distributed: 2007-04-23 |
Word Count: 1064 |
Page Views: 2132
| Votes: 9 |
Rating: 3.89
Redeveloping your website can be a real challenge when you assume
your web designers and developers understand search engine
marketing. Unfortunately this is not usually the case. Most
designers are great at making pretty pictures and developers are
great at making great technology. But pretty pictures and great
technology don't necessarily give you great search engine
results.
How to Prosper Beyond Virtual Real Estate
Written by: Kamau Austin |
Distributed: 2006-10-17 |
Word Count: 1263 |
Page Views: 1593
| Votes: 7 |
Rating: 3.86
We explore virtual real estate (VRE) sites as sites primarily
embedded with Google Adsense(tm) or Yahoo Publisher's Network (YPN)
ads.
|
|
|
Article Reprint Rights
|
This work is
licensed under a
Creative Commons
License
You are not required to show the creative commons license notice when you reprint this work.
|
|
Article Statistics
|
Word Count: 1061
Total Views: 1276
Views This Month: 26
Views Last Month: 28
Article Rating: 3.25 of 5
Votes Cast: 8
Author Overall Stats:
Times Viewed Overall: 59061
Author Rating: 2.27 of 5
Votes Cast Overall: 287
More Articles By Author:
Articles On This Website
Article Title Search:
Widow Search
Right Blinker
Google
Yahoo!
Zuula
Bing
Cuil
Last Distribution Date:
2006-08-02 09:48:00
|
|
|
Home
for Writers
for Publishers
Article Directory
Article Search
Contact
Get RSS Feed
