In Business Since 2001 We accept Paypal
Home Article Directory
More from Bill Platt
"Bill Platt for Book Authors" Youtube Channel
Training for Book Authors
Sites Owned by Bill Platt

 


Today's Google Bots and What They Do

Copyright (c) 2006-2018 , All Rights Reserved
(See This Article in its Original Format.)

Google currently indexes over 8 billion web pages. However, before these pages were placed in the index, they were each crawled by a special spider known as the GoogleBot. Unfortunately, many web masters do not know about the internal workings of this virtual robot.

In fact, Google actually uses a number of spiders to crawl the Web. You can catch these spiders by examining your log files.

This article will attempt to reveal some of the most important Google spiders, their function, and how they affect you as a web master. We'll start with the well-known GoogleBot.

GoogleBot

Googlebot, as you probably know, is the search bot used by Google to scour the web for new pages. Googlebot has two versions, deepbot and freshbot. Deepbot is a deep crawler that tries to folow every link on the web and download as many pages as it can for the Google index. It also examines the internal structure of a site, giving a complete picture for the index.

Freshbot, on the other hand, is a newer bot that crawls the web looking for fresh content. The Google freshbot was implemented to take some of the pressure off of the GoogleBot. The freshbot recalls pages already in the index and then crawls them for new, modified, or updated pages. In this way, Google is better equipped to keep up with the ever-changing web.

This means that the more you update your web site with new, quality content, the more the Googlebot will come by to check you out.

If you'd like to see the Googlebot crawling around your web property more often, you need to obtain quality inbound links. However, there is also one more step that you should take. If you haven't already done so, you should create a Google Sitemap for your site.

Creating a Google sitemap allows you to communicate with Google, telling them about your most important pages, new pages, and updated pages. In return, Google will provide you with some valuable information as well. Google Sitemaps will tell you about pages it was unable to crawl and links it was unable to follow. This allows you to pinpoint problems and fix them so that you can gain increased exposure in the search results.

The next Google bot in our lineup is known as the MediaBot.

MediaBot - used to analyze Adsense pages
useragent: Mediapartners-Google

MediaBot is the Google crawler for Adsense Publishers. Mediabot is used to determine wich ads Google should display on Adsense pages.

Google recommends that webmasters specifically add a command in their robots.txt file that grants Mediabot access to their entire site. To do this, simply enter the following code into your robots.txt file:

User-agent: Mediapartners-Google*
Disallow:

This will ensure that the MediaBot is able to place relevant Adsense ads on your site.

Keep in mind that ads can still be shown on a page if the MediaBot has not yet visited. If that is the case, the ads chosen will be based on the overall them of the other pages on the site. If no ads can be chosen, the dreaded public service announcements are displayed instead.

There is a strong debate over whether or not the MediaBot is giving websites with Adsense an advantage in the search engines.

Even Matt Cutts has confirmed that the Adsense Mediabot is indexing webpages for Google's main index.

He states, "Pages with AdSense will not be indexed more frequently. It's literally just a crawl cache, so if e.g. our news crawl fetched a page and then Googlebot wanted the same page, we'd retrieve the page from the crawl cache. But there's no boost at all in rankings if you're in AdSense or Google News. You don't get any more pages crawled either."

Matt Cutts claims that your website does not get any advantage by using Adsense. However, in my mind, simply getting your site updated in and of itself is an advantage.

This is very similar to Google Analytics, which also promotes a slightly higher degree of spider activity.

Those who run Google Analystics on their site can expect additional spider activity.

However, you certainly shouldn't depend on any of these tools for getting your site indexed. The key to frequent spidering is having quality inbound links, quality content, and frequent updates.

Have images on your site? If so, you have likely been visited by our next Google spider, the ImageBot.

ImageBot - used to crawl for the Image Search
useragent: GoogleBot-Image

The Imagebot prowls the Web for images to place in their image search. Images are ranked based upon their filename, surrounding text, alt text, and page title.

If you have a website that is primarily image based, then you would definitely want to optimize your images to receive some extra Google traffic.

On the other hand, some web sites may not benefit from Google image search. In most cases, the traffic from the Image search engine is very low quality and rarely converts into buyers. Many people are often just looking for images that they can swipe. So, if you want to save some bandwidth, use your robots.txt file to block ImageBot from accessing your image directory.

One of the few exceptions I would make is if you have a site dedicated to downloadable images.

Our final bot is completely dedicated to the Google Adwords program.

AdsBot - Checks Adwords landing pages for quality
useragent: AdsBot-Google

AdsBot is one of Google's newest spiders. This new crawler is being used to analyze the content of advertising landing pages, which helps determing the Quality score that Google assigns to your ads.

Google uses this Quality score in combination with the amount you are willing to bid to determine the position of your ads. Therefore, ads with a high quality score can rank higher even if other advertisers are paying more than you.

This is one of Google's many efforts to ensure that they are delivering the best results to their users.

Can you still block being spidered? Of course, but it will lower your overall Adwords quality score, which could end up lowering the positioning of your ads. If possible, it is best to give AdsBot complete access to your site.

Today's Google bots are becoming more advanced all the time. However, nothing beats relevant, quality, updated content. Deliver that and the search engines will eat it up.


About The Author: Shop Amazon - Top Gift Ideas
Kim Roach is a staff writer and editor for the SiteProNews (http://www.sitepronews.com) and SEO-News (http://www.seo-news.com) newsletters. You can contact Kim at: kim@seo-news.com

This article may be freely distributed without modification and provided that the copyright notice and author information remain intact.

VOTE ON THIS ARTICLE
Needs Work >> 0 - 1 - 2 - 3 - 4 - 5 << Excellent Article

Tell our authors what you think about their article.



Top-Level Category: Business Online Articles

10 Most Recent Articles Written by Kim Roach

Google Gets Personalized
Written by: Kim Roach | Distributed: 2006-12-08 | Word Count: 942 | Page Views: 2365 | Votes: 5 | Rating: 1.40
Traditional algorithmic search engines have reached their peak. Personalized search is a natural and necessary progression for Google and other search engines as well.

7 Online Marketing Tools for the Smart Webmaster
Written by: Kim Roach | Distributed: 2006-12-06 | Word Count: 1144 | Page Views: 3087 | Votes: 6 | Rating: 3.00
As an online marketer, it's hard to wade through all of the misleading information that is running rampant on the Internet. Who do you trust? Where do you go for the very best information in online marketing? Well, this article was written to answer those very questions.

Going Broke on Google Adwords?
Written by: Kim Roach | Distributed: 2006-12-05 | Word Count: 1229 | Page Views: 6592 | Votes: 4 | Rating: 2.25
You've heard the stories. Click fraud has run rampant on Google, Yahoo, and MSN. This is evidenced by the numerous law suits that have been filed.

Creating Your Own Online Radio Station
Written by: Kim Roach | Distributed: 2006-11-27 | Word Count: 1138 | Page Views: 4583 | Votes: 15 | Rating: 1.87
Podcasting has leveled the playing field for people around the world. With just a few tools, you can quickly and easily set up your own talk show on the topic of your choice. All you need is a computer, a microphone, and a passion to share your knowledge with others.

Smart Webmasters are Wasting Online Real Estate
Written by: Kim Roach | Distributed: 2006-11-24 | Word Count: 1307 | Page Views: 2768 | Votes: 6 | Rating: 2.00
Many webmasters are not using their websites' traffic to full potential. The Web is full of wasted advertising space. Chances are that even your website has some virtual wasteland. These are areas that often go unnoticed by even the most experienced marketers, resulting in the loss of thousands of dollars of potential revenue.

Hidden Market Research on Ebay, Amazon, and Others
Written by: Kim Roach | Distributed: 2006-10-12 | Word Count: 1489 | Page Views: 7427 | Votes: 20 | Rating: 1.95
Are you looking to provide interesting content to your visitors? How about popular in-demand products? Do you know what your target customer is really searching for online? No worries. You are about to discover that there are literally thousands of untapped markets just waiting at your fingertips. To get started, let's go on a little virtual journey.

Unique Marketing Strategies
Written by: Kim Roach | Distributed: 2006-10-11 | Word Count: 1629 | Page Views: 5878 | Votes: 14 | Rating: 1.71
As an online marketer I have learned a variety of techniques for attracting traffic. However, sometimes I stumble upon marketing strategies that are truly unique. I urge you to read on.

Web Marketing 2.0
Written by: Kim Roach | Distributed: 2006-10-10 | Word Count: 1645 | Page Views: 4556 | Votes: 8 | Rating: 3.00
If you're like me, then you are probably tired of hearing the word Web 2.0 passed around the net like a buzz word on steroids. Google alone currently shows 126,000,000 results for the search term "web 2.0" at the time of this writing. However, although this term may be overblown, it does have significant influence on the current changes taking place on the Internet. Most importantly, it has many implications for the online marketer.

Franchising Your Online Business
Written by: Kim Roach | Distributed: 2006-09-29 | Word Count: 1833 | Page Views: 3070 | Votes: 15 | Rating: 2.60
Franchises have proven to be much more successful than the traditional small business. Almost 80% of new business start-ups fail each year. Franchising, on the other hand, allows an entrepreneur to open up a new business based upon a proven brand and business model.

Essential Programs For Your Virtual Toolbox
Written by: Kim Roach | Distributed: 2006-09-28 | Word Count: 947 | Page Views: 3992 | Votes: 10 | Rating: 2.60
As an online marketer and webmaster, there are a number of tools that you will want in your virtual toolbox. To get you started, I have scoured the net to find some of the best programs to help webmasters improve their sites, their rankings, and their productivity. Best of all, each one of these tools is free.

All of Author's Articles on this site:

Most Recent "Business Online" Articles

Four Tips For Creating An Effective Authors' Resource Box
Written by: Bill Platt | Distributed: 2013-08-27 | Word Count: 1669 | Page Views: 4001 | Votes: 8 | Rating: 2.38
In article marketing, there are many factors that will affect your articles' ability to promote your website in the most effective manner. In this article, we will briefly discuss a couple those factors, before we dive into how to create an effective Authors' Resource Box, also known as the About The Author Information.

Website Traffic Generation: Winning the Love of the Traffic Fairy
Written by: Bill Platt | Distributed: 2013-08-24 | Word Count: 1149 | Page Views: 4534 | Votes: 6 | Rating: 1.83
Some people still believe that all they have to do is build a website and customers will magically appear on their site... Once they have built their websites, they wait anxiously for the "Traffic Fairy" to come and sprinkle its magic dust on their website too, so that they can make lots of money like other success website owners.

What Kind Of Backlinks Are Best At Helping Your Website In Google?
Written by: Bill Platt | Distributed: 2013-08-21 | Word Count: 439 | Page Views: 5705 | Votes: 8 | Rating: 3.63
I was in the Warrior Forum, and someone asked what kinds of Backlinks are worth pursuing. This is my answer to that query.

Understanding the Mechanics of Effective Article Marketing and Article Syndication
Written by: Bill Platt | Distributed: 2013-08-01 | Word Count: 1399 | Page Views: 2553 | Votes: 3 | Rating: 1.33
Visit several marketing blogs or marketing forums, and you will find dozens of people proclaiming the death of article marketing as an effective marketing tool. Before we start agreeing with them, consider this...

Top 10 Reasons Why Online Marketers Fail With Article Marketing
Written by: Bill Platt | Distributed: 2013-07-31 | Word Count: 1025 | Page Views: 3762 | Votes: 6 | Rating: 2.67
If you've been marketing products online for any length of time, you've likely heard people talk about the power and promise of article marketing for the promotion of your online business. Unfortunately, many online marketers have tried to utilize article marketing to benefit their online business and failed. In this article, I document the top ten reasons why people fail to find success with article marketing... And how to fix your article marketing campaigns...

Article Marketing Tip: Stop Trying To Sell From Within Your Articles
Written by: Bill Platt | Distributed: 2013-07-30 | Word Count: 1229 | Page Views: 2362 | Votes: 3 | Rating: 0.67
The basic premise of article marketing is to present people information that will be important to them, then at the end of the article, present information "about the author" and offer to the reader a means to see how the author can help the reader further. When done well, article marketing continues to be one of the most effective methods of promotion that one could employ online.

The Six Degrees of Internet Marketing
Written by: Bill Platt | Distributed: 2013-07-30 | Word Count: 991 | Page Views: 4383 | Votes: 6 | Rating: 2.33
The theory behind the Six Degrees of Separation is that all people on earth can be connected together in as little as six steps. Now, I know what you might be thinking at this juncture... That sounds fun and interesting, but what does that have to do with Internet Marketing? That is a great question that I am going to answer for you in this article...

How To Win More Eyeballs for Your Offers and Websites
Written by: Bruce Lansing | Distributed: 2013-07-29 | Word Count: 1373 | Page Views: 2174 | Votes: 3 | Rating: 0.33
Most people using search engines on the Internet are looking for answers to questions and solutions to problems. If you can give people what the answers they want, you will be able to position your business as one worth hiring today or tomorrow.

How To Increase Your Rates and Earnings As a Freelance Writer
Written by: Bill Platt | Distributed: 2013-07-29 | Word Count: 1191 | Page Views: 2429 | Votes: 3 | Rating: 0.33
Freelance writers fill a very important role in today's business marketplace. They create the on-demand, customized content that business owners everywhere need to push their businesses forward. In this article, the author explains what freelance writers should do to ensure that employers will pay them well for their services.

Promoting Your Business With Content:: Defining Your Writing Strategy
Written by: Bruce Lansing | Distributed: 2013-07-25 | Word Count: 677 | Page Views: 2103 | Votes: 3 | Rating: 2.00
Many people suggest that you should write articles on the basis of the benefits that people will get from reading your articles. I'd like to suggest that there is a better reason, and that is to write your articles on the basis of what problem people will be able to solve, as a result of reading your articles.

Most Viewed "Business Online" Articles

Attracting Readers to your Book
Written by: Alastair Hall | Distributed: 2006-05-18 | Word Count: 708 | Page Views: 44819 | Votes: 9 | Rating: 2.11
One of the common and obvious ways to market a self-published book is having your own website. The harder part is how to attract your target audience and then convert the sale.

How To Make Visitors Bookmark Your Site
Written by: Cheryl Miller | Distributed: 2006-10-10 | Word Count: 536 | Page Views: 37311 | Votes: 13 | Rating: 2.38
When visitors bookmark your site it is a sign that you have built a great website and are pleasing them. Visitors think highly enough of your site to want to return and possibly tell others about it. You are not only increasing traffic from your existing clientele you are encouraging new traffic by means of viral marketing.

Treat Online 'Guests' With Respect
Written by: Rick Sloboda | Distributed: 2007-07-27 | Word Count: 472 | Page Views: 28773 | Votes: 25 | Rating: 3.76
Does your website show your customers lack of respect? Review the following checklist to find out.

How I Got 70,000 Useless Visitors To My Site In One Day! (One Internet Marketer's Analysis of Social Bookmark Traffic)
Written by: Titus Hoskins | Distributed: 2007-11-06 | Word Count: 1545 | Page Views: 26041 | Votes: 8 | Rating: 2.13
Is social bookmark traffic useless? Is it even worth cultivating for your site? Can it be used from an online marketing perspective? Read to discover one Internet Marketer's analysis of social bookmark/media traffic...

Successful Article Marketers Help Readers Solve Problems
Written by: Bill Platt | Distributed: 2013-07-24 | Word Count: 816 | Page Views: 22276 | Votes: 26 | Rating: 3.04
Every week, I have the opportunity to speak with people about the benefits and challenges of using article marketing to promote an online business. Article marketing is about getting your sales message in front of potential customers and to get links to your website. Many interpret this to mean that an article should directly promote a website within the article, but that approach is wrong and will reduce one's success using this methodology.

Tips For Building Your First Website
Written by: Benny Tsabba | Distributed: 2007-05-30 | Word Count: 1511 | Page Views: 18086 | Votes: 10 | Rating: 2.30
New people are coming online everyday. And many of those people desire to turn their spare time into spare cash. So begins their journey.

The Importance Of Effective Follow-up
Written by: Kate Smalley | Distributed: 2007-01-15 | Word Count: 493 | Page Views: 12565 | Votes: 10 | Rating: 3.20
Marketing experts say that following up with clients is crucial to successful selling because most prospects do not buy the first time, according to Aweber, a leading autoresponder company. People must encounter a marketing message multiple times before making a purchasing decision.

Successful Forum JVs and How To Get Them
Written by: Diana Barnum | Distributed: 2006-06-15 | Word Count: 648 | Page Views: 9087 | Votes: 11 | Rating: 2.18
The forming of common alliances, also referred to as a joint venture (JV), happens nearly every day in the business world. One of the most popular means of linking people up online of so that they can work together on their projects is via forums, where people of all levels of business and expertise post in common threads.

Changing Your Company Name: The Good, The Bad and The Unnecessary
Written by: Marcia Yudkin | Distributed: 2009-07-16 | Word Count: 520 | Page Views: 8999 | Votes: 22 | Rating: 1.91
Wondering whether or not to change your company name? Several reasons for doing this are legitimate. Other reasons should make you stop and reconsider.

Ten Tips For User Friendly Online Registration Forms
Written by: Jim Romanik | Distributed: 2009-02-24 | Word Count: 775 | Page Views: 7668 | Votes: 13 | Rating: 1.00
Have you ever registered or attempted to register for an event and spent half an hour trying to figure out what you needed to do? Or have you used an online registration form that lost your data or was so complicated that it seemed like more work than faxing the form?

Highest Ranked "Business Online" Articles

How to Keep Your Article Content Fresh and Interesting
Written by: Craig Ritsema | Distributed: 2006-08-29 | Word Count: 699 | Page Views: 3806 | Votes: 17 | Rating: 4.18
Webmasters everywhere are constantly on the lookout for fresh and interesting content for their website. Along with these content qualities is the requirement that it be unique. Everyone wants their website to stand out above the others in the search engine results. So what is the trick for you to keep a steady supply of this quality content coming your way?

Is Your Business 'Out Of Site'? Employ The Ultimate Marketing Tool
Written by: Rick Sloboda | Distributed: 2007-01-09 | Word Count: 445 | Page Views: 3834 | Votes: 19 | Rating: 4.16
With the rapid expansion of the digital economy, the web site is conceivably the most powerful marketing and sales tool you can employ. Find out what a web site can do for you.

How To Use Your Race To Get Rich
Written by: Herbert Harris | Distributed: 2009-07-07 | Word Count: 1407 | Page Views: 4491 | Votes: 40 | Rating: 4.05
Amid recent news reports that there continues to be an ongoing and substantial gap between the net worth of Whites and other racial groups, it is easy to be distracted and overwhelmed by the raw data. However, I am immediately reminded of a quote offered by an economic expert from the past... "There are lies, damned lies, and statistics!"

Secrets of Domain Parking - An Innovative and Easier Way to Make Money
Written by: John Khu | Distributed: 2009-02-27 | Word Count: 484 | Page Views: 3891 | Votes: 7 | Rating: 4.00
Using a good domain parking services to monetize expired domain names seems to be a neglected aspect with a number of domain names traders. Domain parking is also an unfamiliar concept to many inexperienced domain name traders. Read this article to better understand domain parking and how it can be of benefit to you.

The Most Important Ingredient for Info Product Success (is You)
Written by: Judy Murdoch | Distributed: 2008-04-15 | Word Count: 981 | Page Views: 4006 | Votes: 13 | Rating: 3.92
Last week I was attending a conference and mentioned to someone I help small business owners create information products. The other person was very friendly until I said "information products." Then the temperature in the room dropped about 10-degrees

How To Build A Website That Will Rank Well In The Search Engines?
Written by: Fred Schebesta | Distributed: 2007-04-23 | Word Count: 1064 | Page Views: 7026 | Votes: 9 | Rating: 3.89
Redeveloping your website can be a real challenge when you assume your web designers and developers understand search engine marketing. Unfortunately this is not usually the case. Most designers are great at making pretty pictures and developers are great at making great technology. But pretty pictures and great technology don't necessarily give you great search engine results.

How to Prosper Beyond Virtual Real Estate
Written by: Kamau Austin | Distributed: 2006-10-17 | Word Count: 1263 | Page Views: 4557 | Votes: 7 | Rating: 3.86
We explore virtual real estate (VRE) sites as sites primarily embedded with Google Adsense(tm) or Yahoo Publisher's Network (YPN) ads.

Why I'm A 'B-Level Marketer!'
Written by: Willie Crawford | Distributed: 2007-07-16 | Word Count: 917 | Page Views: 4428 | Votes: 24 | Rating: 3.83
At a recent Internet marketing seminar, I was sitting around the hotel lobby with approximately 20 other seminar attendees and speakers. The seminar's host was in attendance, and the conversation covered a range of internet marketing topics with extreme frankness. Our proximity to the hotel bar certainly contributed to our 'conviviality.'

The Role of Hunger in Your Business
Written by: Mark Silver | Distributed: 2009-01-21 | Word Count: 1016 | Page Views: 1966 | Votes: 5 | Rating: 3.80
Hunger and fear are good for your business. Not abject hunger and despairing panic, mind you. That's too much. But a little touch of hunger and fear is very useful. I know there are many people out there who would prefer that not to be the case. And I'm one of them. I wish inspiration and love were big enough motivators for action. But for us humans who aren't yet enlightened, they don't seem to work so well.

The Importance Of Creating Resistance In Your Copy
Written by: Bill Platt | Distributed: 2009-09-10 | Word Count: 2009 | Page Views: 2606 | Votes: 5 | Rating: 3.80
"Creating Resistance in Your Copy" is an idea that is alien to most online marketers. After all, online marketers want to believe that everyone should buy their products and services, because the more people buying what is being sold, the more money the marketer will inevitably earn.











Download an eBook today
 
Directory Navigation
Locate By Category:

ALL Categories
Arts & Crafts
Arts & Entertainment
Automotive
Business - Offline
Business - Online
Career
Computers
Education
Family
Finance
Food & Drink
Health & Wellness
Home & Garden
Humor
Internet
Nature & Pets
Real Estate
Religion
Self Improvement
Shopping
Society
Sports & Recreation
Technology
Travel & Leisure
Uncategorized
World Events
Writing & Speaking

Change Number of Results:
50 - 100 - 200 - 500
Article Reprint Rights
Creative Commons License

This work is
licensed under a
Creative Commons
License


You are not required to show the creative commons license notice when you reprint this work.
Article Statistics
Word Count: 1061

Total Views: 12366

Article Rating: 3.25 of 5
Votes Cast: 8

More Articles By Author:



Last Distribution Date:
2006-08-02 09:48:00

Internal ID: #3251





All Articles are Copyright © 2001-2018 of the Defined Authors.

All other material and images on this site are:
Copyright © 2001-2018, ThePhantomWriters.com