Solved: Yandex Bot

JamieIfL · ‎07-01-2018

Hi,

I have a new site, just published and promoted in early June (I've had previous sites with Weebly). The Yandex Bot 3 crawls my site every few hours.

My site is an educational book recommendation site in English, and it has hundreds of pages. While it would be great to think I'm getting my message out to the world, instead, I'm worried that it is web scraping my content. I've installed Google Analytics, but the Google Bot has only been by once, for example.

Is there a way to block this specific bot in a Weebly site? I've looked all over the settings and I know the Yandex Bot does not respect the robots.txt file.

Thanks!

Adam7 · ‎07-01-2018

I don't think it would be possible to block it unless we implement that ourselves. Every search engine is different, and I wouldn't worry that it's trying to scrape content other than what it would use for search results. I would think that the amount of hits would decline, though.

View Best Answer >

Anne5 · ‎07-01-2018

Check your logs for the IP address of the Yandex bot. Then check the IP address to see if it's a legitimate Yandex IP. Yandex is a legitimate search engine. Even if you could block it, it's not a good idea to block Yandex bots - they crawl more times than Google, and they feed other search engines, such as DuckDuckGo, etc.

-- Anne

View Best Answer >

Adam7 · ‎07-01-2018

I don't think it would be possible to block it unless we implement that ourselves. Every search engine is different, and I wouldn't worry that it's trying to scrape content other than what it would use for search results. I would think that the amount of hits would decline, though.

JamieIfL · ‎07-01-2018

Thanks, it's good to know that this could be normal. It's a little weird that it comes by so often, but maybe that's just because I don't have a ton of real visitors yet in comparison.

Anne5 · ‎07-01-2018

Check your logs for the IP address of the Yandex bot. Then check the IP address to see if it's a legitimate Yandex IP. Yandex is a legitimate search engine. Even if you could block it, it's not a good idea to block Yandex bots - they crawl more times than Google, and they feed other search engines, such as DuckDuckGo, etc.

-- Anne

JamieIfL · ‎07-01-2018

Thank you - it looks like it is a real indexing bot. It made me a little nervous that it was so frequent a visitor compared to Google.