The Context (Business Challenge / Problem): A partner running a sports news portal contacted us with a critical issue: the editorial team was entirely unable to publish new articles. A quick infrastructure health check revealed a severe problem: the server’s disk space was at 99% utilization. Further investigation showed that legitimate readers were not the issue. The disk was being aggressively consumed by massive cache files generated by relentless requests from AI bots (GPTBot, ClaudeBot, Applebot) and SEO scrapers (SemrushBot, AhrefsBot). This uncontrolled automated traffic was actively draining disk space, CPU, and RAM, effectively paralyzing the business’s publishing workflow.

The Architecture & Work (Solution): To resolve the incident, we evaluated three mitigation strategies:

  1. Fast and cost-effective: Blocking the bots outright.

  2. Slow and expensive: Deep manual configuration and tuning.

  3. Mid-tier and robust: Deploying an Nginx Reverse Proxy with microcaching and IP rate limiting.

The partner opted for the first solution to instantly restore operability with zero immediate overhead. We analyzed the access logs to pinpoint the exact User-Agents causing the excessive load. We then applied targeted restrictions via the web server configuration and manually purged the bloated cache directories to immediately reclaim system resources.

# 1. Block aggressive bots via .htaccess to instantly drop server load
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GPTBot|SemrushBot|AhrefsBot|Amazonbot|ClaudeBot|Bytespider|Applebot|MJ12bot|DotBot|PetalBot|Baiduspider) [NC]
RewriteRule ^.* - [F,L]

# 2. Purge the overgrown cache files consuming the disk
find /var/www/example.com/cache/ -type f -delete

# 3. Verify that the cache directory is successfully emptied
find /var/www/example.com/cache/ -type f | wc -l
MobaXterm_AMFiDUpjFc.png MobaXterm_GjSssYrRhp.png MobaXterm_aAVAVOT55i.png MobaXterm_sLbghUoVfr.png

The Takeaway (Business Value): By identifying the exact pattern of the automated traffic and enforcing strict access rules, we instantaneously reclaimed over 11GB of disk space. The server began returning 403 Forbidden errors to the scrapers, which immediately and drastically reduced the load on the CPU, RAM, and disk I/O. The partner avoided unnecessary and costly hardware upgrades, and the editorial team was able to resume publishing news within minutes. A follow-up audit is scheduled in two days to verify long-term traffic stabilization.