Disk full

When a disk is full, a server behaves strangely: the database will not start, the website returns errors, files cannot be saved. First find out what has taken up the space — deleting things at random is dangerous.

How much space is left

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        40G   39G  120M 100% /

Look at the line with / — this is the system disk — and at the lines for any additional disks. Use% at 100 means there is no space left.

If there seems to be space but files cannot be created and you get the error No space left on device, check df -i. This is the number of files that fit on the disk. It runs out when there are millions of small files — usually a website cache or PHP sessions.

What is using the space

sudo du -xh -d 1 / | sort -h

This command shows the size of each folder in the root directory, with the largest at the bottom. Then drill down into the largest one: sudo du -xh -d 1 /var | sort -h, and so on. The -x option stops it from crossing over to other disks.

The ncdu program is more convenient: it scans once, and you browse the folders with the arrow keys.

sudo apt install ncdu     # AlmaLinux: sudo dnf install ncdu, from EPEL
sudo ncdu -x /

The largest individual files:

sudo find / -xdev -type f -size +500M -exec ls -lh {} +

What you can usually delete

The systemd journal

journalctl --disk-usage
sudo journalctl --vacuum-size=200M

This keeps the latest 200 MB of the journal. To stop it growing again, set a limit:

sudo mkdir -p /etc/systemd/journald.conf.d
printf '[Journal]\nSystemMaxUse=200M\n' | sudo tee /etc/systemd/journald.conf.d/size.conf
sudo systemctl restart systemd-journald

Old program logs

In /var/log, files ending in a number or .gz — such as syslog.2.gz or access.log.3.gz — are old logs that have already been rotated, and you can delete them. If the current log itself is growing fast, look at what is in it: it is often the same error repeated thousands of times a minute — and that is what needs fixing.

Package cache and old kernels

sudo apt clean && sudo apt autoremove --purge    # Ubuntu, Debian
sudo dnf clean all                                # AlmaLinux
sudo pkg clean -a && sudo pkg autoremove          # FreeBSD

autoremove removes old kernels and packages that nothing needs any more. AlmaLinux itself keeps no more than three kernels.

Docker

docker system df
docker system prune

prune removes stopped containers, unused networks, untagged images and the build cache. With the -a option it also removes all images that no container uses. Data volumes are not touched unless you add --volumes. Another common cause is container logs, which by default grow without limit: how to limit them.

Forgotten copies

A common find is old website archives and database dumps made by hand “just in case”: look in /root, /home, /var/backups and the website folder.

Deleted, but the space was not freed

This means a program still has the file open: it keeps writing to it, and the space stays in use for as long as the program runs. To find such files:

sudo lsof +L1

If the command is not found, install the lsof package. Restart the program from the list, for example with sudo systemctl restart nginx — the space will be freed.

Increasing the disk

If you are constantly short of space, increase the system disk on the Resources tab in your control panel, or add a separate disk for data — Additional disk. The system disk’s partition expands by itself after the server is rebooted. The disk cannot be increased while the server has snapshots — delete them first.