Linux Deleted Logs Still Use Disk Space: Open Files and lsof +L1

If deleting a large log file does not increase free space on a Linux server, first check whether a process still holds the deleted file open. Removing a name with unlink and releasing an open reference with close are separate operations. This article reproduces the behavior with a temporary file, then explains how to investigate with lsof +L1 and why a service may need to reopen its log.

Concept illustration of a file still connected to a running process after its name tag is removed. This is not a filesystem architecture diagram.

Removing the name can leave an open file alive

On Linux, unlink() removes a file's directory entry. Even after its last name is removed, the file remains while a process has it open, until the last open descriptor is closed. If other hard links remain, those names also matter. unlink(2)

A logging process can keep using its existing descriptor rather than resolving the pathname for every write. Creating a new empty file at the same path therefore does not necessarily redirect the process's existing log writes.

Distinguish what df and du measure

df reports space usage for the filesystem containing a specified path. du calculates usage for the specified files and directory tree. An unlinked file still held open by a process can make these observations differ. df(1), du(1)

The following are example diagnostic commands for an operational environment. Align the path and filesystem scope, and check whether permission restrictions omit any entries. A difference between the numbers alone does not prove that deleted open files are responsible.

Reproduce reading a file after unlinking it

Save the following code as deleted_log_demo.py and run it on Linux. It deletes only a file inside the temporary directory it creates; it does not touch existing server logs. It uses /proc/self/fd, so Linux procfs is required.

The observed output was:

The path is gone and the hard-link count is zero, but the open descriptor still reports a logical size of 1 MiB and can read the content. st_size is the logical file length, not a measurement of disk blocks that will be reclaimed. os.fstat() retrieves information through an open descriptor rather than a pathname. Python os.fstat documentation

/proc/PID/fd contains an entry for each descriptor the process holds open. The example confirmed a (deleted) suffix on its own entry's target. After the inner with block ends, it also verifies that accessing that descriptor number fails with EBADF. proc_pid_fd(5)

Identify the process with lsof +L1

If lsof is installed and you have sufficient inspection permissions, the following selects open files with a link count below one. +L1 selects open files that have been unlinked. lsof(8)

Inspect the process name, PID, FD, file type, device and inode, and name together. When investigating regular log files, start with REG entries. Multiple descriptors or processes can refer to the same file, so do not simply sum the size in every row. SIZE/OFF can represent size or offset depending on file type and options; check its meaning and header.

This command is a documentation-based operational example. No user production server was inspected and no service was restarted for this validation.

Resolve the log reference rather than deleting more files

After confirming the cause, prefer the service's supported log-reopen or log-rotation procedure. If a restart is necessary, first assess service impact and recovery steps, then perform a controlled restart. Do not assume that sending a particular signal makes every program reopen its logs.

Check again that references to the unlinked file have disappeared, then recheck filesystem free space. Reopening one process's log may leave another process holding the same file. Repeated deletion or blindly truncating a descriptor path should not be the default recovery procedure.

Validation used Python 3.12.14 on Linux x86_64. Six checks passed: pathname disappearance, zero link count, preserved logical size, the deleted marker, reading after unlink, and descriptor invalidation after close. This did not validate filesystem-wide reclaimed block counts, a real service's log rotation, or container and remote-filesystem behavior.