Website slowed down by a flood of requests

Sometimes a website suddenly slows down and the log shows a huge number of requests — thousands a minute. Often this is not malicious, just an over-eager bot downloading the whole site. Either way, the approach is the same: find out who is sending the requests and limit them.

Who is sending the requests

# the most active addresses
sudo awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head

# the most active programs (bots identify themselves here)
sudo awk -F'"' '{print $6}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head

For Apache, the log is /var/log/apache2/access.log (Ubuntu, Debian) or /var/log/httpd/access_log (AlmaLinux).

Blocking an address

# ufw (Ubuntu, Debian)
sudo ufw insert 1 deny from 203.0.113.5

# nginx — in the server or http block
deny 203.0.113.5;

In nginx, after making the change, run sudo nginx -t && sudo systemctl reload nginx. On FreeBSD, use a table in pf: add the lines table <blocked> persist and block in quick from <blocked> to /etc/pf.conf, then add the address with sudo pfctl -t blocked -T add 203.0.113.5.

Limiting the request rate

To stop a single address from sending more than 10 requests per second, use this in nginx:

http {
    limit_req_zone $binary_remote_addr zone=perreq:10m rate=10r/s;
    limit_req_status 429;

    server {
        location / {
            limit_req zone=perreq burst=20 nodelay;
            # ...
        }
    }
}

burst=20 allows short bursts — a browser loads a page as a batch of requests. To limit the number of simultaneous connections from an address instead, see this separate article.

Bots

You can keep out an unwanted bot by the name it gives, which you can see in the second list above, with a line in the nginx server block:

if ($http_user_agent ~* (BadBot|OtherBot)) { return 403; }

Do not block search engines: most of them, such as Google and Bing, let you set the crawl rate for your site in their webmaster tools.

If the server does not respond at all — even over SSH — log in through the console in your control panel and contact our support team through the contact form: we will look at what is reaching the server.