Please help i checked robots.txt file ,Google has...

rickbeaker · ‎01-08-2021

I Checked my site on google search console and discoved google had ignored 25 of my pages and has not crawled it . when i checked robots.txt file this was what i got

Sitemap: http://b************/sitemap.xml

User-agent: NerdyBot
Disallow: /

User-agent: *
Disallow: /ajax/
Disallow: /apps/

how can i change my robots.txt file so google can crawl the excluded pages and this has affected my rankings alot. please help

seicolegwr · ‎01-09-2021

@rickbeaker

The robots file is automatically generated based on the make-up of your website but you have no access to it. That Nerdybot disallow is standard for all Weebly websites, it has no effect on Google or any other of the major search engines crawling your site. Ajax and apps disallows are also standard fare for websites.

The line User-agent: * allows all search engines to crawl your site. So, have you checked to see if the option to 'hide your site from search engines' is disabled for the whole website and/or spectific pages?

Gary

Please help i checked robots.txt file ,Google has blogged most of my pages from crawling.