Scholar

Why robots.txt file disallow: / issue ????

Hi,

 

I have created lots of websites in weebly but suddenly i know about search engine is disallow my all websites that means my all websites are not indexing and crawling in search engine.

 

I think there is some issues in weebly - robots.txt file....because robots.txt default format is as below:

 

Allow indexing of everything

 

User-agent: *

Disallow:

 

That means all webpages and blogs are allow indexing and crawling in search engine.

But, Weebly has consider below format, that means totally wrong from SEO perspective.

 

Disallow indexing of everything

 

User-agent: 
Disallow: /

That means no any webpages and blogs are not allow for indexing and crawling in search engine.

 

I am totally disappointed with this issue please solve this issue ASAP.

 

here are my weebly websites 

 

Alibaba Clone | Yelp Clone | Review Script

 

 

6 REPLIES
Community Manager

Re: Why robots.txt file disallow: / issue ????

Hey @sandraendo! All you need to do is go to Settings > SEO for each of your sites, and disable the "hide from search engines" option. Re-publish after doing that and it will update your robots.txt file for you automatically.

- Adam
Senior Community Specialist
LBZ
Explorer

Re: Why robots.txt file disallow: / issue ????

Hi Adam,

I disallowed the hiding option yesterday and I can find the website in google now, but title and description are still disallowed.

The title showed by google is just the url of the website (instead of the title I typed) and in the description field I read:

"A description for this result is not available because of this site's robots.txt"

 

Shall I wait for more or shall we consider this a problem?

 

Please advice.

Thanks

LBZ
Explorer

Re: Why robots.txt file disallow: / issue ????

website is fincip.io

Community Manager

Re: Why robots.txt file disallow: / issue ????

It might take a little longer for Google to update that information.  Have you set up your site with Google Webmaster tools?

 

https://www.weebly.com/seo/search-console

 

Google has a "Fetch as Google" option which might help speed things up:

 

https://support.google.com/webmasters/answer/6065812?hl=en

- Adam
Senior Community Specialist
Scholar

Re: Why robots.txt file disallow: / issue ????

Yes, This is all done by me on GWT...

 

But my question is if you remove just disallow: / [slash]  than solve this all issues and search engine is easily indexing our websites...

N/A

Re: Why robots.txt file disallow: / issue ????

I believe this is an issue and can weebly look at it or atleast provide a better answer? I believe something becomes an issue if it is recurring across many users