x

Why robots.txt file disallow: / issue ????

Hi,

I have created lots of websites in weebly but suddenly i know about search engine is disallow my all websites that means my all websites are not indexing and crawling in search engine.

I think there is some issues in weebly - robots.txt file....because robots.txt default format is as below:

Allow indexing of everything

User-agent: *

Disallow:

That means all webpages and blogs are allow indexing and crawling in search engine.

But, Weebly has consider below format, that means totally wrong from SEO perspective.

Disallow indexing of everything

User-agent: 
Disallow: /

That means no any webpages and blogs are not allow for indexing and crawling in search engine.

I am totally disappointed with this issue please solve this issue ASAP.

here are my weebly websites 

Alibaba Clone | Yelp Clone | Review Script

1,848 Views
Message 1 of 7
Report
6 REPLIES 6
Square

Hey @sandraendo! All you need to do is go to Settings > SEO for each of your sites, and disable the "hide from search engines" option. Re-publish after doing that and it will update your robots.txt file for you automatically.

1,839 Views
Message 2 of 7
Report

Hi Adam,

I disallowed the hiding option yesterday and I can find the website in google now, but title and description are still disallowed.

The title showed by google is just the url of the website (instead of the title I typed) and in the description field I read:

"A description for this result is not available because of this site's robots.txt"

Shall I wait for more or shall we consider this a problem?

Please advice.

Thanks

1,782 Views
Message 2 of 7
Report

website is fincip.io

1,781 Views
Message 2 of 7
Report
Square

It might take a little longer for Google to update that information.  Have you set up your site with Google Webmaster tools?

https://www.weebly.com/seo/search-console

Google has a "Fetch as Google" option which might help speed things up:

https://support.google.com/webmasters/answer/6065812?hl=en

1,777 Views
Message 2 of 7
Report

Yes, This is all done by me on GWT...

But my question is if you remove just disallow: / [slash]  than solve this all issues and search engine is easily indexing our websites...

1,763 Views
Message 2 of 7
Report

I believe this is an issue and can weebly look at it or atleast provide a better answer? I believe something becomes an issue if it is recurring across many users
957 Views
Message 2 of 7
Report