What Is AhrefsBot? User-Agent, IPs & How to Block (or Allow) It

If you see AhrefsBot hammering your logs or showing up in firewall alerts, you’re looking at Ahrefs’ own SEO crawler. It’s noisy, it’s powerful, and it’s usually harmless - as long as you control it properly.

This guide explains in plain language:

What AhrefsBot and AhrefsSiteAudit are
The AhrefsBot user-agent, IPs, and how to verify real vs fake traffic
How to block or slow AhrefsBot with robots.txt, server rules, or a firewall
How to get AhrefsBot to crawl your site if it’s not showing in Ahrefs

TL;DR (Quick Answer)

AhrefsBot is Ahrefs’ SEO crawler. It powers Ahrefs tools and the Yep.com search engine.
It’s considered a “good bot”: it respects robots.txt, Crawl-delay, and uses public IP ranges.
To block AhrefsBot everywhere via robots.txt:
```
User-agent: AhrefsBot
Disallow: /
```
To slow AhrefsBot instead of blocking it:
```
User-agent: AhrefsBot
Crawl-delay: 10
```
To confirm real AhrefsBot traffic, the IP should be in Ahrefs’ published ranges and reverse-DNS to a hostname ending with ahrefs.com or ahrefs.net.

AhrefsBot at a Glance

Typical info people search for: ahrefs bot, ahrefsbot, ahrefs robot, http ahrefs com robot.
Here’s the quick spec sheet that answers those queries up front.

Field	Value
Bot name	AhrefsBot
Type	SEO crawler (web index for Ahrefs + Yep.com)
Main user-agent token	`AhrefsBot`
Example full UA	`Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)`
Bot URL	`http://ahrefs.com/robot` (what you see in logs as `+http ahrefs com robot`)
Respects robots.txt?	Yes – `Allow`, `Disallow` and `Crawl-delay` supported
Public IP ranges?	Yes, Ahrefs publishes them for whitelisting/blocking

What Is AhrefsBot (Ahrefs Bot)?

AhrefsBot is the primary web crawler run by Ahrefs. It constantly scans the web and stores:

URLs and internal links
Titles, headings, and other on-page data
Anchor texts and linking pages (for backlinks)

That index powers tools like:

Site Explorer (backlinks, referring domains, organic traffic)
Keywords Explorer (keyword ideas, SERP data)
Site Audit (technical SEO checks, via a related crawler)
The Yep.com search engine

In other words: the better AhrefsBot can crawl your site, the more accurate your data will be in Ahrefs.

What Is AhrefsBot For? (Use Cases)

If you’re wondering “what is ahrefs bot for?”, it mainly exists to collect SEO data, not to attack your site.

AhrefsBot is used to:

Build a backlink index
- Discover which sites link to which
- Measure link quality & authority
- Detect new links and lost links over time
Map content and keywords
- Extract titles, headings, and anchor text
- Understand what each URL is about to support keyword tools
Feed Yep.com search results
- Same crawl data reused for a privacy-focused search engine

So if you or your agency rely on Ahrefs, you generally want AhrefsBot to see your site.

AhrefsBot vs AhrefsSiteAudit (Two Different Bots)

Ahrefs actually runs two main crawlers:

AhrefsBot – the global crawler
- Constantly crawls the public web
- Builds the shared link/content index used across Ahrefs
AhrefsSiteAudit – the audit crawler
- Runs when you start a Site Audit project
- Uses a separate user-agent and can have custom speed settings
- Sometimes allowed to ignore robots.txt for your own site if you explicitly enable that in Ahrefs Webmaster Tools (for troubleshooting blocked sections)

Knowing the difference helps you block or limit each separately if needed.

Ahrefs Bot Names & User-Agents

1. Main AhrefsBot user-agent

User-agent name: AhrefsBot

Example UA string you’ll see in logs:

Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)

Key things from this string:

AhrefsBot is the token you target in robots.txt (User-agent: AhrefsBot)
The +http://ahrefs.com/robot/ part explains why people search “http ahrefs com robot” – that’s the contact URL in the UA string

2. AhrefsSiteAudit user-agent (“audit bot”)

User-agent name: AhrefsSiteAudit

Example UA:

Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit)

You’ll mainly see this when you run Site Audit on your own site in Ahrefs.

How AhrefsBot Crawls Your Site (Robots.txt & Crawl-Delay)

Ahrefs states that both AhrefsBot and AhrefsSiteAudit:

Obey robots.txt allow/disallow rules
Support Crawl-delay to slow their requests
Don’t invent random URLs – they follow links they find
Use only GET requests (they shouldn’t trigger destructive actions)

In practice:

If robots.txt disallows AhrefsBot for a path, it will not crawl there.
If you set Crawl-delay, it waits at least that many seconds between requests.
If your server starts returning many 4xx/5xx errors, AhrefsBot backs off and crawls more slowly.

This makes AhrefsBot a “polite” crawler compared to most scrapers.

Is AhrefsBot Safe? Should You Block It?

Is AhrefsBot a “bad bot”?
Generally no:

It’s a well-known SEO crawler, recognized as a legitimate bot in many security tools and crawler lists.
It respects standard controls (robots.txt, Crawl-delay, IP disclosure).

The real question is: should you block AhrefsBot?

Reasons to allow AhrefsBot

You get fresh backlink and keyword data in Ahrefs.
Agencies and partners using Ahrefs can see accurate data for your site.
It’s easy to throttle instead of blocking outright.

Reasons to block or limit AhrefsBot

Very small or resource-limited sites where crawl bursts noticeably slow things down.
Highly sensitive infrastructure or internal projects that should never appear in third-party SEO tools.
Environments where any non-essential crawler is banned by policy.

Default recommendation:
For most public websites, don’t fully block AhrefsBot. Use:

Crawl-delay to slow it, and/or
Path-specific Disallow rules (e.g. /admin/, /private/)

…instead of Disallow: /.

How to Block AhrefsBot in robots.txt

If you really want an ahrefs bot disallow, the standard way is via robots.txt.

⚠️ Blocking AhrefsBot means no updated data in Ahrefs for your domain. That affects you and anyone else using Ahrefs to analyze your site.

1. Fully block AhrefsBot

User-agent: AhrefsBot
Disallow: /

This tells AhrefsBot not to crawl any URLs on your site. Ahrefs’ documentation explicitly states this is how to stop their bot.

2. Block both AhrefsBot and AhrefsSiteAudit

User-agent: AhrefsBot
Disallow: /

User-agent: AhrefsSiteAudit
Disallow: /

Use this if you don’t want any Ahrefs crawler touching your site at all.

3. Block only specific paths

If you only want certain sections hidden:

User-agent: AhrefsBot
Disallow: /admin/
Disallow: /private/
Disallow: /internal-reports/

This is usually better than nuking the whole site from Ahrefs’ index.

How to Limit (Throttle) AhrefsBot Instead of Blocking It

If the concern is server load, it’s usually enough to slow AhrefsBot down.

1. Use `Crawl-delay` in robots.txt

Ahrefs explicitly supports the Crawl-delay directive.

Example: limit AhrefsBot to roughly one URL every 10 seconds:

User-agent: AhrefsBot
Crawl-delay: 10

Combine it with partial disallows:

User-agent: AhrefsBot
Crawl-delay: 10
Disallow: /admin/
Disallow: /staging/

Note: Crawl-delay is non-standard, but Ahrefs documents support for it, even though Google ignores it.

2. Let Ahrefs auto-throttle on errors

If your site responds with many 4xx/5xx codes, Ahrefs reduces crawl speed for that domain automatically.

That can work as a short-term safety net during outages, but it’s not a permanent control - because those errors also hurt users and other bots (like Googlebot).

3. Adjust speed for AhrefsSiteAudit

For AhrefsSiteAudit specifically:

Ahrefs sets a default max URLs/min
Verified site owners can manually lower the crawl rate from inside the tool for their own audits

That’s the safest way to control the “Ahrefs audit bot” without blocking global crawling.

Blocking AhrefsBot by IP / Firewall (Last Resort)

Sometimes robots.txt isn’t enough (e.g. on very sensitive infrastructure or where you need strict enforcement). Then you can block AhrefsBot by IP.

1. Get the official Ahrefs IP ranges

Ahrefs publishes the IP ranges and individual addresses they use for crawling.

Typical flow:

Fetch the current list from Ahrefs’ help center.
Keep it updated (they occasionally add or adjust ranges).

2. Confirm IPs with reverse DNS

To make sure a request is from the real AhrefsBot:

Do a reverse DNS lookup on the IP
Check the hostname ends with ahrefs.com or ahrefs.net

If it doesn’t, it may be a fake bot spoofing the AhrefsBot user-agent.

3. Example: Block Ahrefs IPs in Apache (.htaccess)

(Adjust ranges to the latest ones from Ahrefs; these are just structural examples.)

# Block AhrefsBot by IP (example ranges – replace with current ones)
<RequireAll>
    Require all granted
    Require not ip 192.0.2.0/24
    Require not ip 198.51.100.0/24
</RequireAll>

4. Example: Nginx server-level block

# Block AhrefsBot by IP (example)
geo $block_ahrefs {
    default 0;
    192.0.2.0/24       1;
    198.51.100.0/24    1;
}

server {
    if ($block_ahrefs) {
        return 403;
    }

    ...
}

Important: IP blocking is harder to maintain than robots.txt and can break Ahrefs data for your site completely. Use it only when you really need that level of control.

AhrefsBot IP: How to Check Real vs Fake Traffic

Because user-agents are easy to fake, it’s common for scrapers to call themselves AhrefsBot. To filter real AhrefsBot from impostors:

Pull Ahrefs’ official IP ranges from their help article or API.
For suspicious IPs, run reverse DNS and confirm the hostname ends with ahrefs.com or ahrefs.net.
Build firewall rules that:
- Whitelist or rate-limit only verified Ahrefs IPs
- Treat fake “AhrefsBot” traffic like any other random bot

This is much safer than relying on the user-agent string alone.

“Ahrefs Audit Bot Blocked” & Other Crawl Issues

If Ahrefs shows 0 crawled pages, or you see logs like “Ahrefs audit bot blocked”, common causes are:

1. Robots.txt blocks AhrefsBot

If your robots.txt contains rules like:

User-agent: AhrefsBot
Disallow: /

or similar for AhrefsSiteAudit, Ahrefs cannot crawl your site.

Fix:

User-agent: AhrefsBot
Allow: /

User-agent: AhrefsSiteAudit
Allow: /

Or remove the disallow rules for these bots.

2. Robots.txt is missing or returns errors

If https://yourdomain.com/robots.txt returns 5xx or times out, AhrefsBot may treat that as a sign of instability and crawl cautiously. If your important pages also return 4xx/5xx, Ahrefs can’t index content at all.

Fix:

Ensure robots.txt returns HTTP 200 quickly
Make sure main pages return 200 as well

3. Server, WAF, or CDN is blocking AhrefsBot

Common blockers:

ModSecurity rules
WordPress/other security plugins
WAF/CDN configurations (Cloudflare, Sucuri, etc.) that flag AhrefsBot’s IPs or user-agent

Fix:

Whitelist AhrefsBot’s user-agent and Ahrefs IP ranges in your firewall/CDN rules.
If you’re on managed hosting and can’t edit rules, open a ticket and ask them to follow Ahrefs’ bot docs.

Once these issues are fixed, AhrefsBot should resume crawling on its next scheduled pass.

How to Get AhrefsBot to Crawl Your Site More Reliably

If you’re specifically asking “how do I get Ahrefs bot to crawl my site?”, use this checklist.

1. Allow AhrefsBot in robots.txt

At minimum, make sure you don’t block it.

Example:

User-agent: AhrefsBot
Allow: /

Remove any Disallow: / lines that target AhrefsBot or AhrefsSiteAudit.

2. Check your site with Ahrefs’ robots checker

Ahrefs provides tools to test whether their bots can access your URLs. Start from the ahrefs.com/robot page and look for their robots status / access checkers.

This tells you quickly whether robots.txt, HTTP status codes, or other issues are blocking crawling.

3. Remove IP or firewall blocks

Whitelist AhrefsBot and the official IP ranges in:
- Hosting firewall
- Security plugins
- CDN/WAF (Cloudflare, etc.)

If a provider blocks Ahrefs and you can’t change it directly, send a support request referencing Ahrefs’ bot documentation and ask them to allow AhrefsBot.

4. Fix basic HTTP errors & performance issues

Ensure important pages respond with 200 OK, not 4xx/5xx
Avoid extended periods where the whole site returns 5xx (maintenance, outages)
Keep robots.txt fast – slow or failing robots.txt can cause crawl issues across many bots

5. Verify your domain in Ahrefs Webmaster Tools (Optional)

If you verify ownership in Ahrefs Webmaster Tools, you can:

See how AhrefsBot is crawling your site
Run Site Audit on demand
Tune the audit crawl speed (for AhrefsSiteAudit)

Verification doesn’t guarantee more frequent global crawling, but it gives you visibility and control.

FAQ About AhrefsBot

Is AhrefsBot a search engine crawler?

Partly, yes.

Its main role is to fuel Ahrefs’ SEO platform (link + content data).
The same crawl data is also used by the Yep.com search engine.

What is the official AhrefsBot user-agent?

A typical user-agent string looks like:

Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)

The important part for robots.txt is just AhrefsBot.

How often does AhrefsBot crawl my site?

It depends on:

Your site’s size and authority
How often your content changes
Any crawl limits (robots.txt Crawl-delay, disallows, IP blocks)
How stable your site’s responses are (few 5xx/timeout issues)

In general, AhrefsBot is one of the most active SEO crawlers, scanning large portions of the web on a frequent schedule.

How do I block AhrefsBot only from one directory?

In robots.txt:

User-agent: AhrefsBot
Disallow: /private-directory/

You can add multiple lines for additional folders.

What happens if I block AhrefsBot?

Ahrefs’ tools will no longer get fresh data about your site:

Backlink counts and referring domains may freeze or become outdated
Your pages may be missing from certain Ahrefs reports
Agencies relying on Ahrefs to audit or report on your site will see incomplete data

If that’s acceptable for you, blocking is safe. Otherwise, prefer throttling and path-level rules.

What Is AhrefsBot? User-Agent, IPs & How to Block (or Allow) It

TL;DR (Quick Answer)

AhrefsBot at a Glance

What Is AhrefsBot (Ahrefs Bot)?

What Is AhrefsBot For? (Use Cases)

AhrefsBot vs AhrefsSiteAudit (Two Different Bots)

Ahrefs Bot Names & User-Agents

1. Main AhrefsBot user-agent

2. AhrefsSiteAudit user-agent (“audit bot”)

How AhrefsBot Crawls Your Site (Robots.txt & Crawl-Delay)

Is AhrefsBot Safe? Should You Block It?

Reasons to allow AhrefsBot

Reasons to block or limit AhrefsBot

How to Block AhrefsBot in robots.txt

1. Fully block AhrefsBot

2. Block both AhrefsBot and AhrefsSiteAudit

3. Block only specific paths

How to Limit (Throttle) AhrefsBot Instead of Blocking It

1. Use Crawl-delay in robots.txt

2. Let Ahrefs auto-throttle on errors

3. Adjust speed for AhrefsSiteAudit

Blocking AhrefsBot by IP / Firewall (Last Resort)

1. Get the official Ahrefs IP ranges

2. Confirm IPs with reverse DNS

3. Example: Block Ahrefs IPs in Apache (.htaccess)

4. Example: Nginx server-level block

AhrefsBot IP: How to Check Real vs Fake Traffic

“Ahrefs Audit Bot Blocked” & Other Crawl Issues

1. Robots.txt blocks AhrefsBot

2. Robots.txt is missing or returns errors

3. Server, WAF, or CDN is blocking AhrefsBot

How to Get AhrefsBot to Crawl Your Site More Reliably

1. Allow AhrefsBot in robots.txt

2. Check your site with Ahrefs’ robots checker

3. Remove IP or firewall blocks

4. Fix basic HTTP errors & performance issues

5. Verify your domain in Ahrefs Webmaster Tools (Optional)

FAQ About AhrefsBot

Is AhrefsBot a search engine crawler?

What is the official AhrefsBot user-agent?

How often does AhrefsBot crawl my site?

How do I block AhrefsBot only from one directory?

What happens if I block AhrefsBot?

Related Blog Posts

What CMS do you use?

1. Use `Crawl-delay` in robots.txt