Key takeaways
- AhrefsBot powers Ahrefs' data and the Yep search engine. It obeys robots.txt and crawl-delay.
- To ban it, disallow the AhrefsBot user-agent in robots.txt; for a hard block, use a firewall rule.
- Blocking it only hides your site from Ahrefs' data. It doesn't affect Google.
What AhrefsBot Is
AhrefsBot is the web crawler operated by Ahrefs. According to Ahrefs, it powers the database behind its SEO platform and Yep, its search engine. It identifies itself with the user-agent token AhrefsBot and follows robots.txt rules, including crawl-delay. Ahrefs runs a second crawler, AhrefsSiteAudit, for its Site Audit tool.
Both are listed by Cloudflare as verified bots. Ahrefs publishes its crawler details and IP ranges on its official bot page, ahrefs.com/robot.
Should You Block It?
Usually not. Blocking AhrefsBot doesn't change your Google rankings; it only stops Ahrefs from updating its data about your site. Reasons people do block it:
- Server load on a small or slow host.
- Hiding backlink and content data from competitors who use Ahrefs.
- Keeping private or staging sections out of third-party indexes.
If load is the only concern, slowing the bot down is usually better than banning it.
Option 1: Ban AhrefsBot With robots.txt
Add these lines to the robots.txt file at the root of your domain:
User-agent: AhrefsBot
Disallow: /To keep Ahrefs' site audits working on your own site while blocking the main crawler, allow the audit bot separately:
User-agent: AhrefsBot
Disallow: /
User-agent: AhrefsSiteAudit
Allow: /Ahrefs notes that it can take some time for the bot to pick up changes to robots.txt.
Option 2: Block Only Certain Sections
User-agent: AhrefsBot
Disallow: /admin/
Disallow: /staging/
Disallow: /client-portal/This keeps your public pages visible in Ahrefs while hiding private areas.
Option 3: Slow It Down With Crawl-Delay
User-agent: AhrefsBot
Crawl-delay: 10The value is the minimum number of seconds between requests. Ahrefs says crawl-delay is followed for HTML pages but can't be followed the same way when it renders JavaScript, because a page's images, scripts and stylesheets load together. If you still see bursts in your logs, combine crawl-delay with caching.
Option 4: Hard Block at the Firewall
robots.txt is a request, not a lock. Legitimate bots follow it, but anything can spoof the AhrefsBot user-agent. For a hard block, create a firewall or CDN rule (for example in Cloudflare) that blocks the bot by its verified status or by Ahrefs' published IP ranges. To block impostors while allowing the real bot, allow only requests from Ahrefs' published IPs that claim to be AhrefsBot.
How to Check It Worked
- Look for
AhrefsBotin your server or CDN logs after a few days. - Confirm your robots.txt has no typos in the user-agent name; small errors cause bots to ignore the rule.
- If you use Ahrefs yourself, run its robots checker on your domain.
What Blocking Won't Do
Blocking AhrefsBot won't remove existing data from Ahrefs immediately, won't stop other SEO tools' crawlers, and won't affect Googlebot or Bingbot. If your real problem is crawl load from many bots, look at caching and rate limiting across the board. If you'd like a second pair of eyes on your crawl setup, my technical SEO work covers it.