Scholar

Create and upload a real robot.txt

This weebly idea of a simple on off button is childish and oversimplified when someone wants to creat and upload an ACTUAL file....please tell me how, or why this is not possible.

There seem to be thousands of people looking for this, the most BASIC of web functionality.

10 REPLIES
Community Manager

Re: Create and upload a real robot.txt

What is it you want to add to your robots.txt file, @healinghands? It's mostly useful for telling search engines what not to index and where your sitemap file is. Your sitemap is automatically created, and pages of your site are removed or added from it based on what you choose to hide from search engines.

- Adam
Community Manager
Scholar

Re: Create and upload a real robot.txt

Hi there,

As I mentioned my main INDEX page is currently set to DISALLOW and NO it is not checked off as hidden, please do not offer that as the fix. 

It seems that everyone from weebly who responds does not really read the question, but simply offers up a common answer based on maybe skimming a word, in this case robot.txt

It would seem that one way to fix the issue would be to modify that file

Community Manager

Re: Create and upload a real robot.txt

I figured it out, @healinghands - you have a page called "home - temp" which has its permalink set to index.html. That page is hidden from search engines so it's adding index.html to your robots.txt file. If you remove the permalink from that and re-publish it should fix it for you.

- Adam
Community Manager
Scholar

Re: Create and upload a real robot.txt

You are a genuis ! THANK YOU

I have the other disallow pages as headers only in the Navigation, to lead to submenu items...is it OK they are disallow?  does that mean the pages under them do not get indexed?>? or do they?

 

Thanks

 

Community Manager

Re: Create and upload a real robot.txt

That shouldn't hurt anything since those aren't actual pages and have no content on them. Smiley Happy

- Adam
Community Manager
Scholar

Re: Create and upload a real robot.txt

Here is the reason one would want to create a real robots.txt:

All the category pages and archive pages inheret the page title from the parent page.

That means they all have the same title/description, which is bad for seo.

There is really no reason to index these pages. Being able to have a robots.txt would solve this problem.

 

Or weebly could make the page titles unique.

Scholar

Re: Create and upload a real robot.txt

yes i have been reading that as well.

Scholar

Re: Create and upload a real robot.txt

I dont want search engine to index any of my pictures. How to do that without editing robots.txt ?

Community Manager

Re: Create and upload a real robot.txt

I don't think there's a way to do that without adding something to the robots.txt file, @Vladimir83, which you're already aware we don't allow direct editing of. Unfortunately, I don't see a way to do this unless our engineers add support for it. If you'd like us to do that you're welcome to post it on our Vote on Features board.

 

- Adam
Community Manager