This weebly idea of a simple on off button is childish and oversimplified when someone wants to creat and upload an ACTUAL file....please tell me how, or why this is not possible.
There seem to be thousands of people looking for this, the most BASIC of web functionality.
What is it you want to add to your robots.txt file, @healinghands? It's mostly useful for telling search engines what not to index and where your sitemap file is. Your sitemap is automatically created, and pages of your site are removed or added from it based on what you choose to hide from search engines.
As I mentioned my main INDEX page is currently set to DISALLOW and NO it is not checked off as hidden, please do not offer that as the fix.
It seems that everyone from weebly who responds does not really read the question, but simply offers up a common answer based on maybe skimming a word, in this case robot.txt
It would seem that one way to fix the issue would be to modify that file
I figured it out, @healinghands - you have a page called "home - temp" which has its permalink set to index.html. That page is hidden from search engines so it's adding index.html to your robots.txt file. If you remove the permalink from that and re-publish it should fix it for you.
You are a genuis ! THANK YOU
I have the other disallow pages as headers only in the Navigation, to lead to submenu items...is it OK they are disallow? does that mean the pages under them do not get indexed?>? or do they?
That shouldn't hurt anything since those aren't actual pages and have no content on them.
Here is the reason one would want to create a real robots.txt:
All the category pages and archive pages inheret the page title from the parent page.
That means they all have the same title/description, which is bad for seo.
There is really no reason to index these pages. Being able to have a robots.txt would solve this problem.
Or weebly could make the page titles unique.