The Analytics SEO Crawler
If you're seeing this page, you are probably looking at your web server logs, finding a bunch of page requests from a user agent called "Curious George", and wondering who is putting all this load on your site.
The Analytics SEO crawler carries out a number of tests on various websites on behalf of our customers. Some tests (e.g. for sitemaps, custom 404 pages, robots.txt) are carried out daily, and every week we will carry out a full crawl of our customers' websites. These tests help our customers to make their web sites more accessible to crawlers as part of their search engine optimisation work. There's more detail about exactly what we do here.
If your site is being accessed by our crawler, it means one of two things:
- Your business is being managed by the Analytics SEO system. This could be because you have signed up directly, or if your SEO work is handled by an agency, they could be using our platform. Alternatively you could have signed up through one of our many partner companies (e.g. Scoot/TouchLocal)
- A customer of the Analytics SEO platform has added your site as one of their competitors. In this case, we will continue to carry out the daily tests, but will not do the weekly crawl of your entire site
We make every effort to keep the impact of our crawler on your system to a minimum. The full site crawl respects robots.txt directives, and we limit the rate we send queries to your servers. The daily tests are designed to get the data we need in as few requests as possible. However, if your systems are adversely affected by the activity of our crawler, please do get in touch with us using the Feedback form.