.png&w=3840&q=75)
Websites & Hosting4 min
How to Block Malicious User Agents Using .htaccess
NS
NameSilo Staff9/3/2026
Share
To block malicious user agents and bad bots, you can add security directives to your server's .htaccess file. Use the RewriteCond and RewriteRule commands to identify the specific User-Agent string of the aggressive bot, and instruct the server to return a 403 Forbidden error, instantly severing the connection before it consumes your CPU resources.
What Is a User-Agent String?
Every request hitting your server, whether from a human's browser or an automated script, includes a User-Agent header identifying what's making the request. A real visitor's browser sends something like Mozilla/5.0 ... Chrome/120.0. A bot often identifies itself too: GPTBot/1.0, SemrushBot, Bytespider.
This string is the digital fingerprint you use to tell the server exactly who to let through and who to turn away.
Why It Matters: Preserving CPU and Bandwidth
Aggressive scrapers don't politely browse your site the way a human does. They hammer every URL they can find, often hundreds or thousands of requests per hour, whether it's an AI training crawler harvesting content, an SEO tool auditing your backlinks, or a vulnerability scanner probing for weaknesses.
One real-world example: a site owner tracking their logs found Bytespider (ByteDance's crawler) alone accounted for 36.58% of all requests over a two-week period, more traffic than every legitimate search engine combined. That's CPU, memory, and bandwidth spent on a bot instead of real visitors.
Many bots ignore robots.txt entirely. Even among ones that check it, current research shows roughly 89% of domains now disallow GPTBot specifically, yet unwanted AI traffic keeps climbing regardless, since compliance is voluntary.
Locating the Threat
Before writing any rule, confirm exactly which bot is responsible.
In cPanel, go to Metrics → Raw Access Logs, download the current log, and search for repeated entries hitting the same paths in rapid succession. The User-Agent string appears at the end of each log line, in quotes. Note the exact name before writing your block rule; guessing leads to mistakes.
Implementation Steps
Add this to your site's .htaccess file:
RewriteEngine OnRewriteCond %{HTTP_USER_AGENT} (BadBotName|AnotherBadBot) [NC]RewriteRule .* - [F,L]Replace BadBotName with the exact string from your logs. Separate multiple bots with the pipe character. The [NC] flag makes matching case-insensitive; [F,L] returns a 403 Forbidden and stops processing further rules.
Test before trusting it: run curl -I -A "BadBotName" https://yourdomain.com/ from a terminal. A working rule returns 403 Forbidden immediately.
Common Mistakes
Writing overly broad match patterns: Blocking the literal word "bot" catches Googlebot, Bingbot, and every other legitimate crawler your SEO depends on, wiping out organic visibility in one careless line. Always match the full, specific bot name, never a generic substring.
Assuming user-agent blocking is unbeatable: Sophisticated scrapers increasingly spoof their identity, claiming to be a regular Chrome browser instead of announcing themselves honestly. Current research found most AI agent traffic doesn't declare itself properly, meaning user-agent rules stop the bots that are honest about who they are, while more evasive traffic may still get through. Pair this with rate limiting or IP-range blocking for bots known to hide.
The Honor System vs the Brick Wall
robots.txt is a polite request. It tells bots where you'd prefer they not go, and well-behaved crawlers comply voluntarily. Nothing stops a bot from reading it and ignoring it anyway.
.htaccess is different: it's enforced at the server level. A blocked request never reaches your application code at all, no PHP runs, no database query fires. The bot doesn't get a choice to comply or not; the door is physically shut.
What This Means for You
NameSilo Hosting includes cPanel File Manager access for editing .htaccess directly, along with Raw Access Logs for identifying exactly which bots are consuming your resources. Building a fresh site to protect from day one? Register a domain to get started.
Frequently Asked Questions
How do I block bad bots from my website?
Add RewriteCond rules matching the bot's User-Agent string in .htaccess.
What is a User-Agent string?
Text a browser or bot sends identifying itself with each request.
How do I block an IP address in htaccess?
Use Require not ip inside a RequireAll block for Apache 2.4.
Can I block AI scrapers using htaccess?
Yes, matching names like GPTBot, ClaudeBot, CCBot, or Bytespider.
What happens if a bot ignores robots.txt?
Nothing stops it there; .htaccess enforcement is the actual barrier.
How do I block SemrushBot or AhrefsBot?
Add their exact User-Agent name to your RewriteCond pattern.
Why is my server CPU usage so high?
Often aggressive scrapers hitting many URLs repeatedly; check Raw Access Logs.
Does NameSilo hosting offer bot protection?
Standard infrastructure protections apply; .htaccess rules add manual control.
.png&w=2048&q=75)
NameSilo StaffThe NameSilo staff of writers worked together on this post. It was a combination of efforts from our passionate writers that produce content to educate and provide insights for all our readers.
More articleswritten by NameSilo
.png&w=3840&q=75)
.png&w=3840&q=75)