Slow server: how to find the cause

A slow server is almost always limited by one of three things: the CPU, memory or disk. A few commands will show which one, and which process is to blame.

Load: load average

$ uptime
 10:42:01 up 12 days,  3:07,  1 user,  load average: 3.52, 2.10, 1.05
$ nproc
2

The three numbers are the average load over 1, 5 and 15 minutes: how many processes, on average, were running or waiting for their turn. On Linux this also includes processes waiting for the disk. Compare the numbers with the number of cores (nproc): a load of 2 on two cores means the server is fully busy, and 4 means tasks are queuing.

If the first number is higher than the third, the load is rising; if it is lower, the load is falling. In the example above, 3.52 on two cores means the server is overloaded, and it is getting worse.

top and htop: what is causing the load

top is available on every system. htop is easier to use — it has bars for each core, search and scrolling. It is already installed on Ubuntu; on other systems:

sudo apt install htop    # Debian
sudo dnf install htop    # AlmaLinux, from EPEL
sudo pkg install htop    # FreeBSD

The %Cpu(s) line in top shows what the CPU time is being spent on:

Memory

$ free -h
               total        used        free      shared  buff/cache   available
Mem:           3.8Gi       2.9Gi       150Mi        20Mi       1.0Gi       0.7Gi
Swap:          1.0Gi       0.4Gi       0.6Gi

The key column is available: how much memory can still be given to programs. A small free figure is normal: Linux uses spare memory as a disk cache and hands it back as soon as it is needed. The problem is when available is close to zero and the amount of swap in use is growing.

To see what is using the memory, run ps aux --sort=-rss | head — the RSS column is in kilobytes. On FreeBSD, use top -o res.

OOM killer: a program stopped by itself

When memory runs out completely, the Linux kernel forcibly terminates the largest process — most often the database or PHP. From the outside it looks like this: “the website went down and MySQL stopped for no reason”. To check:

sudo journalctl -k | grep -i -E 'out of memory|killed process'

-k shows kernel messages since the last boot; to see the previous boot, add -b -1. On AlmaLinux the journal of earlier boots is not kept by default, so look in /var/log/messages instead. If you find matching lines, the server does not have enough memory. What you can do:

• configure your programs to use less: fewer PHP-FPM processes, smaller database buffers;
• add a swap file — a reserve for short peaks;
• add more memory — see below.

Disk

A high wa value in top, and a high load while the CPU is idle, mean that everything is waiting for the disk. To see which processes read and write the most, use pidstat from the sysstat package:

sudo apt install sysstat    # or dnf install sysstat
pidstat -d 1                # reads and writes by process, every second
iostat -x 1                 # disk load: the %util column

Common causes are backups running at peak times, database queries without indexes, logs that are too detailed, and swapping — when memory is short, the system has to write it out to disk. If the disk is also full, see Out of disk space.

The server is idle but the website is slow

If there is hardly any load but the website responds slowly, the problem is in the website itself: slow database queries, or calls to external services that are slow to respond. If the load is created by requests from outside — bots, or a surge of visitors — see A website slowed down by a surge of requests.

When to add resources

If the load is constantly higher than the number of cores, or memory is always running out, and your programs are already tuned, add CPUs or memory. In your control panel, open the server and go to the Resources tab.

CPU and memory can only be changed on a server that is switched off, so choose when to apply the change: immediately — the server shuts down, the change is made and it starts again, which takes about a minute — or the next time it is switched off, when you do that yourself.

Charts of CPU, memory and disk usage for previous days are on the Load tab: they show when the problems began.