In one of my previous posts I described what could be the cause for the above problem.
Now Google has its own definition for this. Whenever Googlebot crawls a site it first looks for the robots.txt file to get the information about the blocked URL’s.
If your server was down at that instant, Googlebot just defers (postpones) the crawl on your site since its unsure of the instructions given in the robots.txt file(Remember,Googlebot seriously respects robots.txt file).
The bot crawls your site the next time your server is UP or next time when your site crawl turn comes up.
In case there is no robots.txt file found on the root directory,the Googlebot crawls the site as if every URL is asked to be crawled.
Visitors who read this article also read:
- Sitemap.xml error URL timeout:robots.txt timeoutI recently came across this error when I logged onto my webmaster tools.I searched the web to find out the...
- xml-sitemap error, make sure the file exists and is writableIf you got the above message when trying to implement the XML sitemap, then you have the solution here. I’m...
- Google sitemap guidelinesEver since Google introduced sitemaps,it set very strict guidelines in the making of a sitemap. For your sitemap (which ever...
- Webmaster community in OrkutOrkut is a social networking site that most of us use here in India and no wonder that is the...
- Google advanced search optionsYou might have experienced this, you search for something but you get bogged down by seeing millions of sites that...
- Redirect your domain to the www versionPreviously I’ve described the Importance of having www in your Blog’s URL and the role that it plays in optimizing...