Tourist

Robots.txt

Hello,

 

Somehow, a text was added into robots.txt and I'm unable to edit it. The text stops certain crawlers to crawl my site.

 

User-agent: NerdyBot
Disallow: /

I need to remove this, unfortunalty support keeps sending me canned messges so no use.

 

Thanks 

13 REPLIES
Mayor

Re: Robots.txt

Hi, there!

 

Have you submitted your sitemap to Google Webmaster tools? That might fix this issue for you.

 

Thanks!

 

Erin

Weebly Community Manager

Re: Robots.txt

Weebly automatically generates the robots.txt file for you, and it's not possible to edit it. Note that changes you make in the Weebly editor determine what gets added to the robots.txt file. For example, on the Pages tab in the Advanced section, if you check the box for "Hide this page from search engines" it will add that page to the bottom of the list. 

Weebly has made the decision to block NerdyBot, and it's not possible to change that.

So, you should see something like: 

Sitemap: https://www.your-domain.com/sitemap.xml

User-agent: NerdyBot
Disallow: /

User-agent: *
Disallow: /ajax/
Disallow: /apps/

If there is nothing after Disallow: /apps/ it means that nothing else on your site is being blocked.

___________________________
jeffreytkastner.com - Web Designer
Explorer

Re: Robots.txt

Thanks for this information, Jeffrey.  I've recently become aware that FreeFind is now not able to search my site here (SeattleYASS.weebly.com).  It worked fine for years, but some time before May 2016 it became blocked.

 

FreeFind tells me I should put the following lines in robots.txt:

user-agent: FreeFind
allow: /

 

Can something like this be done, or is FreeFind blocked for some reason?  Is there anything I can do about this?

Community Manager

Re: Robots.txt

You won't be able to directly edit your site, though you can do this:

 

1. Go to Settings > SEO

2. Disable the option to hide your site from search engines

3. Re-publish

 

That will update your robots.txt file so it doesn't block search engines in general (including FreeFind).

- Adam
Senior Community Specialist
Explorer

Re: Robots.txt

Thanks, Adam, that did it.  I don't know how "hide from search engines" got turned on -- I certainly didn't do it!  When I turn that off FreeFind can do its indexing and all is well.

 

Best regards,

Gerrit

Community Manager

Re: Robots.txt

I'm not sure either, but I'm glad to hear it's working for you!

- Adam
Senior Community Specialist

Re: Robots.txt - not editable in Weebly... What now?

I use Google Merchant as well as Weebly and Google MErchant is telling me it cannot crawl my images.  How do I fix that?  Thanks in advance for your help.

Community Manager

Re: Robots.txt - not editable in Weebly... What now?

Your robots.txt file and domain DNS records look to be set up ok, @autoprofishopUS. What error is Google Merchant giving in specific, and what URL are you using?

- Adam
Senior Community Specialist

Re: Robots.txt - not editable in Weebly... What now?

Hi, Adam-

In replying to your message, I just realized what I think I did wrong....

I may have used the wrong image URLs.  I listed the URLs which hav the weebly domain associated (which is what you get when you right click and copy the image location from WITHIN the "edit site" mode. 

image link
https://www.weebly.com/uploads/7/4/3/1/74316665/s644759774199853307_p1_i22_w320.png
 
 
 
 
 
 
 
 
 

What I should have done is close "edit site," and returned to my site's shopping page and copied the image link which would be associated then to MY URLinstead of weebly's...

http://www.autoprofishop.us/uploads/7/4/3/1/74316665/s644759774199853307_p1_i22_w450.png

(This may be a different product, but it's part of the same batch of images).

 

YOU ARE A GENIUS!  THANK YOU!  LOL (that was kinda easy, eh?)