x

Network unreachable: robots.txt unreachable and Network unreachable: robots.txt unreachable

Hi there,

In Google Webmasters I've started getting the subjected errors in Search Console for my sitemap.xml file

The first error contains the following information:

We were unable to crawl your Sitemap because we found a robots.txt file at the root of your site but were unable to download it. Please ensure that it is accessible or remove it completely.

The second error contains no further information.  However, both have the following in the description of the error:

We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit

This only started happening towards the backend of last week.  Prior to that, no issues were experienced at all.  I've checked with my domain name provider and everything looks fine from their end, and I've even logged support on the Google Webmasters forums but as yet haven't received any useful info.

Any assistance that can be provided on here would be greatly appreciated.

4,464 Views
Message 1 of 11
Report
10 REPLIES 10

The site is www.palsc.com.au and is totally accessible if you go to the website itself. There don't appear to be any DNS issues that I can see...
4,463 Views
Message 2 of 11
Report

It looks like you have a few pages blocked on your robots.txt file. You'll want to check "Settings" > "SEO" and make sure you have "hide on search engines" turned off. Also under "pages" > click on the page on the left > SEO settings > make sure at the bottom it doesn't have the checkbox checked for hide from search engines. 

Disallow: /under-11s.html
Disallow: /juniorpresday.html
Disallow: /under-10s.html
Disallow: /under-9s.html
Disallow: /under-8s.html
Disallow: /under-17s.html
Disallow: /under-7s.html
Disallow: /under-16s.html
Disallow: /under-6s.html
Disallow: /under-15s.html
Disallow: /under-14s.html
Disallow: /under-13s.html
Disallow: /under-12s.html

  

4,455 Views
Message 2 of 11
Report

SEO definitely does not have "hide in search engines" ticked, so that's all good.

As for the pages that are hidden, these are purposefully hidden as they are not yet ready for consumption by visitors to the site.  These have also been blocked in the robots.txt file for a number of weeks without any issue.  This particular problem only appeared late last week.

4,452 Views
Message 2 of 11
Report
Square

I don't see any reason why Google would have trouble access either file; like you mentioned your DNS are fine, and I can view both files myself without any trouble. It's possible it was just some kind of temporary issue that prevented Google from viewing them, and it hasn't tried since.

4,436 Views
Message 2 of 11
Report

Hi Adam,

I've tried resubmitting the sitemap.xml in the Google Search Console and continue to get the same errors.  For a temporary issue it has now been on-going fo r at least 5 days.

Any other thoughts you may have on possible causes?  The interwebs seem very dark on this particular problem...

4,432 Views
Message 2 of 11
Report

Hello OlorinFiresky!

It sounds like you may be waiting on a recrawl there, for Google's system to check the site again. That can take up to a few weeks, though it's hard to say more from here. The files all seem in order and accessible from this end, however.

4,421 Views
Message 2 of 11
Report

Hi Queso,

If it was as simple as a recrawl then surely if I were to do a Fetch as Google it should result in success because I'm forcing the issue.  Yet I get the following error and have done since this problem first appeared...

Temporarily unreachable
 
This stack exchange thread seems to suggest it is something to do with the host server not responding in time or Fetch as Google cancelled the fetch because too many consecutive requests were made to the server for different URLs...
 
Thoughts?
4,395 Views
Message 2 of 11
Report
Square

What is the date on when the fetch error was detected? I've found that Google will keep the error listed with other server errors even though it might be months old. You might want to use the checkboxes below to select and delete any old server errors.

4,392 Views
Message 2 of 11
Report

Did you ever get an actionable answer to your question? I'm having same problem and it seems to be caused by Weebly directing the boots to editmysite as a resource instead of to my actual domain (I don't have a free site, so it should not be trying to crawl the Weebly parent site).

1,697 Views
Message 2 of 11
Report

I resolved this myself in the end.  I had mutiple domain options listed and this was causing the issue.  So I had https://domainname.com, https://www.domanname.com, http://domainname.com, and http://www.domainname.com.  Once I removed all but http://www.domainname.com the problem went away.

963 Views
Message 2 of 11
Report