x

How modify the file robot.txt???

Hi, is there the posibility to modify the file robot.txt??

Now my file is like this:


User-agent: *
Disallow: /

and i want permit all...

Thanks

12,815 Views
Message 1 of 44
Report
1 Best Answer

Best Answer

Hi Tony!

I know you found the answer to your question, but just so anyone else reading is clear...

Weebly automatically creates a blank robots.txt file for your site. If you look at your robots.txt file (www.yourdomainhere.com/robots.txt), you'll see text similar to this:

User-agent: *
Disallow: /ajax/
Disallow: /apps/

The user-agent: * means that the rules in this file applies to all robots. The "Disallow: /ajax/" and "Disallow: /apps/ " lines mean that robots cannot index any files you've uploaded to your store to sell (this is a default setting, so you'll see this line even if you're not using the Weebly store features). If nothing else is listed, it means that nothing else on your site is disallowed. If there were anything being blocked, it would show up after the second "disallow" but as long as that is blank, none of your pages are blocked and search engines can find your site.

If you want to change this and block search engines from indexing your site, go to Settings > SEO and select "Hide site from search engines." Then Save and Publish.

The robots.txt file for a site that is completely hidden from search would look like this:

User-agent: *
Disallow: /

Let us know if there are any other questions!

View Best Answer >

12,804 Views
Message 3 of 44
Report
43 REPLIES 43

Resolved

12,422 Views
Message 2 of 44
Report

Best Answer

Hi Tony!

I know you found the answer to your question, but just so anyone else reading is clear...

Weebly automatically creates a blank robots.txt file for your site. If you look at your robots.txt file (www.yourdomainhere.com/robots.txt), you'll see text similar to this:

User-agent: *
Disallow: /ajax/
Disallow: /apps/

The user-agent: * means that the rules in this file applies to all robots. The "Disallow: /ajax/" and "Disallow: /apps/ " lines mean that robots cannot index any files you've uploaded to your store to sell (this is a default setting, so you'll see this line even if you're not using the Weebly store features). If nothing else is listed, it means that nothing else on your site is disallowed. If there were anything being blocked, it would show up after the second "disallow" but as long as that is blank, none of your pages are blocked and search engines can find your site.

If you want to change this and block search engines from indexing your site, go to Settings > SEO and select "Hide site from search engines." Then Save and Publish.

The robots.txt file for a site that is completely hidden from search would look like this:

User-agent: *
Disallow: /

Let us know if there are any other questions!

12,805 Views
Message 3 of 44
Report

Thanks for clearing that up, I had the same question Smiley Happy
12,338 Views
Message 4 of 44
Report

Hi there,

the cituation is exactly same in my site, and I do not know why when I google my site name, it says it is robots.txt file so blabla

I do want my website to be visible and open for search engines. 

Any help?

12,266 Views
Message 5 of 44
Report

Hey phenol, message me which site we're discussing, and I'll take a look!

EDITED TO ADD: I advised to add Google Webmaster Tools code, as described here:

http://hc.weebly.com/hc/en-us/articles/201723433-Verify-Site-with-Search-Engines

12,263 Views
Message 6 of 44
Report

Hi,

If you can do something for me too, as i have problem with robot.txt file, it is blocked on Google webmaster tool.

(robots.txt file disallow the search engines access to some parts of your website).I would like to be searchable on google.

Or explain here how to fix problem with.

Thanks,

Nenad

12,193 Views
Message 7 of 44
Report
Square


@Nenad wrote:

Hi,

If you can do something for me too, as i have problem with robot.txt file, it is blocked on Google webmaster tool.

(robots.txt file disallow the search engines access to some parts of your website).I would like to be searchable on google.

Or explain here how to fix problem with.

Thanks,

Nenad


Open your site in th editor, go to Settings > SEO, and disable the option to hide the site from search engines. After that, re-publish your site and you should be good to go, @Nenad. Smiley Happy

12,188 Views
Message 7 of 44
Report

Thank you!

Solved!

Tags (1)
12,182 Views
Message 7 of 44
Report
Square


@Nenad wrote:

Thank you!

Solved!


Not a problem! Glad to help. Robot Very Happy

12,182 Views
Message 7 of 44
Report

that sounds easy but mine is not solved cuz: this is what i see 

User-agent: NerdyBot
Disallow: /

I turned button ON. SEO

and i made sitemap on google search console and i go to varvy seo tool and it says that google bots not allowed.

and the above is what i see after searching /robots.txt   on my url.

can you pleaseee help? how can i get the robots.txt to say  

User-agent: NerdyBot
Disallow:
5,732 Views
Message 7 of 44
Report

Hello! Sorry for any confusion.

That particular setting would be for disallowing NerdyBot, and only that entity, from indexing the site. There wouldn't be a way to remove that setting through Weebly. The slash there won't affect your indexing elsewhere, however, so you shouldn't see any impact from it.

5,732 Views
Message 7 of 44
Report

Hi, i have similar question of robot.txt

i go to SEO, select 'off' on the 'hide the side from search enqine', saved and published.

but my website stil showing 'side description is not allowed because of robot.txt'

how to solve my problem..

thanks

3,733 Views
Message 7 of 44
Report

Hey there lvtheresa!

Once your Robots.txt is updated, you may need to wait for the search engine to re-index the site, in order for it to see the updated information and change the search. Sorry for the confusion there!
3,733 Views
Message 7 of 44
Report

After you took Adam's suggestion to go to Settings > SEO, and disable the option to Hide the site from search engines, how long did it take before google started crawling your entire site again?  It's been over a week since I took Adam's advice and my google search console still shows:

Warnings
Url blocked by robots.txt.
Sitemap contains urls which are blocked by robots.txt.

   

My home page has been indexed, but not the other 7 pages.

Thank you for your help, Adam!  

5,904 Views
Message 7 of 44
Report
Square

It can take a little while for Google to update the preview shown in search results.  Have you set up Webmaster Tools with your site? You can try using the Fetch as Google option to get Google to clear this quicker.

https://support.google.com/webmasters/answer/6065812?hl=en

5,902 Views
Message 7 of 44
Report

@Adam wrote:

It can take a little while for Google to update the preview shown in search results.  Have you set up Webmaster Tools with your site? You can try using the Fetch as Google option to get Google to clear this quicker.

https://support.google.com/webmasters/answer/6065812?hl=en


Hi Adam, yes, I have a Webmaster account and I have just done what you suggested and had Googe Fetch and Render each page.  Let's hope!

Tags (1)
5,897 Views
Message 7 of 44
Report
Square

Awesome! That should definitely speed things up for you.  

5,896 Views
Message 7 of 44
Report

Hi Adam,

The "hide the site from search engine" was already disable but I am still seeing "Disallow: /" in robots.txt

Please help.

Thanks!

Imee

1,943 Views
Message 7 of 44
Report
Square

What site are you editing, @imeelibra? I don't see any sites in the account tied to your Community profile.

1,942 Views
Message 7 of 44
Report

Hi I am having an issue when I search my site in Google it says it doesn not contain more info because of the robot txt setting. I want to make sure that info about my site is showing in a Google search not simply the link.

7,939 Views
Message 21 of 44
Report