Somehow, a text was added into robots.txt and I'm unable to edit it. The text stops certain crawlers to crawl my site.
User-agent: NerdyBot Disallow: /
I need to remove this, unfortunalty support keeps sending me canned messges so no use.
Have you submitted your sitemap to Google Webmaster tools? That might fix this issue for you.
Weebly Community Manager
Weebly automatically generates the robots.txt file for you, and it's not possible to edit it. Note that changes you make in the Weebly editor determine what gets added to the robots.txt file. For example, on the Pages tab in the Advanced section, if you check the box for "Hide this page from search engines" it will add that page to the bottom of the list.
Weebly has made the decision to block NerdyBot, and it's not possible to change that.
So, you should see something like:
Sitemap: https://www.your-domain.com/sitemap.xml User-agent: NerdyBot Disallow: / User-agent: * Disallow: /ajax/ Disallow: /apps/
If there is nothing after Disallow: /apps/ it means that nothing else on your site is being blocked.
Thanks for this information, Jeffrey. I've recently become aware that FreeFind is now not able to search my site here (SeattleYASS.weebly.com). It worked fine for years, but some time before May 2016 it became blocked.
FreeFind tells me I should put the following lines in robots.txt:
Can something like this be done, or is FreeFind blocked for some reason? Is there anything I can do about this?
You won't be able to directly edit your site, though you can do this:
1. Go to Settings > SEO
2. Disable the option to hide your site from search engines
That will update your robots.txt file so it doesn't block search engines in general (including FreeFind).
Thanks, Adam, that did it. I don't know how "hide from search engines" got turned on -- I certainly didn't do it! When I turn that off FreeFind can do its indexing and all is well.
I use Google Merchant as well as Weebly and Google MErchant is telling me it cannot crawl my images. How do I fix that? Thanks in advance for your help.
Your robots.txt file and domain DNS records look to be set up ok, @autoprofishopUS. What error is Google Merchant giving in specific, and what URL are you using?
In replying to your message, I just realized what I think I did wrong....
I may have used the wrong image URLs. I listed the URLs which hav the weebly domain associated (which is what you get when you right click and copy the image location from WITHIN the "edit site" mode.
What I should have done is close "edit site," and returned to my site's shopping page and copied the image link which would be associated then to MY URLinstead of weebly's...
(This may be a different product, but it's part of the same batch of images).
YOU ARE A GENIUS! THANK YOU! LOL (that was kinda easy, eh?)