Scholar

Robots.txt error on Google Console

Hi there, I am a beginner with the SEO side of things. 

On the Google webmasters site I am getting this message on the first page of Search Console

 

Severe health issues are found in your property. Check property health

Is robots.txt blocking important pages?
Some important page is blocked by robots.txt.

 

Sitemap: http://www.vinylupdate.co.nz/sitemap.xml

User-agent: NerdyBot
Disallow: /

User-agent: *
Disallow: /ajax/
Disallow: /apps/

 Could someone please let me know how to fix this? Many thanks

8 REPLIES
Scholar

Re: Robots.txt error on Google Console

I'm guessing this isnt really and issue and I should just ignore those errors...

Community Manager

Re: Robots.txt error on Google Console

The two folders that are blocked from indexing are resources used by your site that you don't want indexed (scripting and so forth), so that's normal to see. I always find it strange that Google gives a message like that when there are valid reasons to not want something indexed. Thanks for asking, though!

- Adam
Senior Community Specialist
Explorer

Re: Robots.txt error on Google Console

I have got the same problem. Please help me to solve it. When i google the website, the description of the website does not show and i get a message that says, "A description for this result is not available because of this site's robots.txt"

Community Manager

Re: Robots.txt error on Google Console

Was there a period of time after you pointed your domains to Weebly that your sites weren't yet published? It sounds like Google indexed them prior to having the sites live. Google should update this information automatically, although it's hard to say when they will.

- Adam
Senior Community Specialist
Explorer

Re: Robots.txt error on Google Console

Did you also get a message on the google search page that said, "A description for this result is not available because of this site's robots.txt"

Scholar

Re: Robots.txt error on Google Console

I'm apparently having a similar problem with a robot.txt file - and cannot find it.  It appears to be blocking Google's access to my first page, which is where my Nabble forum is embedded, so Google isn't finding the content of the forum, but does all the other pages of the site.

 

I've checked the SEO settings and I do not have the page hidden from search engines.  I've checked the header and footer code, to no avail.  But here's what Google's Analytics say.  What am I missing?

 

Capture of Crawl Errors.JPG

Gary Lewis
Employee Moderator

Re: Robots.txt error on Google Console

Definitely not an expert, but does this help out at all? https://support.google.com/webmasters/answer/6153277?hl=en

Scholar

Re: Robots.txt error on Google Console

Thanks for the reply.  I've actually read that, albeit subsequent to posting.  It provided some info, as did the Weebly SEO guide.  At this point it looks like the problem is with Nabble, not Weebly as the /robots.txt file shows the page to be open, meaning not blocked.  So, something on the page isn't working, and about all that is on there is a piece of code Nabble gave me to embed.

 

So, I'm trying to get help from Nabble.  But, if you know what to do please tell me.

Gary Lewis