CCBot is the crawler used by Common Crawl, which maintains an open repository of web crawl data. A request from a crawler is automated activity; it does not establish that a person read the page or guarantee inclusion in Google Search.
Historical record
- IP address:
44.211.239.1. - Country recorded by the site: United States.
Recorded user-agent:
CCBot/2.0 (https://commoncrawl.org/faq/)This historical address is not a current allowlist. The user-agent and country recorded in a log do not, on their own, authenticate the source of a request.
Verification and crawl controls
Common Crawl warns that other clients can claim to be CCBot. Its official CCBot documentation publishes current IP ranges and a reverse-DNS verification example with a forward lookup back to the original address. Consult the current guidance; it notes a limitation for reverse DNS over IPv6.
The documentation also explains how to request that CCBot not crawl a site using its own group in robots.txt. Such rules communicate crawl preferences; they are not authentication or a way to secure private files.
When reviewing activity, check request frequency, paths and HTTP responses. Do not decide whether an entry is legitimate solely by comparing it with the old IP above.
Read more about what web crawlers do.
Comments (0)
Comments are shown in their original language.
No comments have been published yet. Be the first to join the conversation.