Bot Detection: How to Find Scrapers and Abusive Automation

Last verified on Nov 8, 2026.

Sooner or later automated clients reach every public endpoint: crawlers you want, scrapers you do not and scripts that probe login or checkout. The ones that hurt are the quiet ones. They stay under any single alarm, copy your catalog or directory and show up in your bill as extra traffic.

OWASP files this under OAT-011 Scraping. It describes a scraper that may read every accessible path and parameter value on pages and APIs, may use fake or compromised accounts and may run continuously or in periodic sweeps. The same page lists LLM and AI scrapers among the names for it.

What the evidence looks like

Bots leave a different shape than people. One client asks for many distinct URLs, in order, at a steady pace, with no asset requests.

203.0.113.7 - - [08/Nov/2026:10:02:11 +0000] "GET /directory?page=411 HTTP/1.1" 200 18233 "-" "python-requests/2.31.0"
203.0.113.7 - - [08/Nov/2026:10:02:12 +0000] "GET /directory?page=412 HTTP/1.1" 200 18102 "-" "python-requests/2.31.0"
203.0.113.7 - - [08/Nov/2026:10:02:13 +0000] "GET /directory?page=413 HTTP/1.1" 200 18377 "-" "python-requests/2.31.0"

That is an example, not a real client. A rotating scraper looks different: the same path walk, but each address makes only a few requests. You only see it by grouping.

How to detect it

1. Log client address, network or ASN, method, path, status, user agent, session or account id and response size for every request.
2. Compute per-client features over a sliding window of 5 to 15 minutes: request count, distinct paths, share of requests that hit data endpoints, share of asset requests and error ratio.
3. Group by something an attacker cannot rotate cheaply: session, account, ASN or a request signature such as user agent plus header order.

SELECT asn,
       count(*) AS requests,
       count(DISTINCT client_ip) AS ips,
       count(DISTINCT path) AS distinct_paths,
       round(100.0 * count(*) FILTER (WHERE path LIKE '/assets/%') / count(*), 1) AS asset_pct
FROM access_log
WHERE ts > now() - interval '15 minutes'
GROUP BY asn
HAVING count(*) > 2000 AND count(DISTINCT path) > 500
ORDER BY requests DESC;

A network that makes thousands of requests across hundreds of paths with almost no asset traffic is a scraper far more often than a crowd of visitors.

You can add managed signals on top. Cloudflare gives each request a bot score from 1 to 99 and runs heuristics, JavaScript detections for headless browsers and machine learning engines. AWS WAF has rate-based rules that count requests per aggregation key over an evaluation window. If you log those verdicts, you can join them to your own features and see where the two disagree.

Thresholds and pitfalls

Start with a rate threshold per client, then tune from your own distribution. Take the 99th percentile of requests per session on a normal day and flag clients that are several times above it for a sustained window. A burst does not count. A steady rate for 10 minutes does.

Pitfalls that cause false positives:

• Offices, schools and mobile carriers put many real users behind one address. Prefer session or account keys when you have them.
• Search crawlers are bots you want. Do not trust the user agent. Google documents verifying Googlebot with a reverse DNS lookup, a check that the domain is googlebot.com, google.com or googleusercontent.com, and a forward lookup that returns the same IP.
• Your own monitors, partners and mobile apps may call data endpoints at high rates. Allow-list them by credential, not by address.
• Seasonality matters. A product launch or a newsletter send raises everyone's rate, so compare against the same hour last week.

What to do when it fires

Decide whether the traffic is harmful before you block it. If it is, apply a rate limit or a challenge at the edge first, since a hard block tells the operator to switch addresses. Require authentication for data that should not be public, and cap page size and pagination depth. robots.txt states your preference but, as RFC 9309 says, it is not access authorization.

Related guides: credential stuffing and other authentication anomalies often come from the same networks, and cloud spend anomalies can follow when heavy scraping raises your traffic bill. If your edge is on Cloudflare or AWS, the Cloudflare and AWS pages list the logs you can ship for this analysis.