Prevent Overlapping Linux Batch Jobs with flock
Use flock to serialize Linux batch jobs and reproduce how deleting a locked file lets another process lock a different file at the same path.
If a Linux batch starts again before its previous run finishes, have every run acquire an exclusive flock on the same file before doing the work. Keep the lock file at a stable path and use the lock state to decide whether to run. Independent Python processes demonstrate contention, release, and the hazard of deleting a locked file.
Two cyan process modules and an amber padlock attached to a file
File existence and lock ownership differ
Checking for a file before creating it leaves a race in which another process can intervene. flock requests a lock on an open file. Combining an exclusive lock with LOCK_NB makes the request fail immediately when another process holds a conflicting lock. Python exposes this through fcntl.flock. Linux flock(2) · Python fcntl documentation
The lock is associated with an open file description. Closing every file descriptor that references that lock releases it. An empty file can remain at the path, and the next run can still acquire the lock. The path can stay in place after the job finishes. Linux flock(2)
Use a stable lock file
In a shell, run a command with flock -n /var/lock/my-job.lock /opt/jobs/my-job. Choose a directory where the batch account can open the file, and use the same path for every invocation. Decide whether the scheduler treats the contention exit code as a failure or a skipped run. If child processes inherit the lock file descriptor, they can extend the lock lifetime. Check process lifetimes when the job starts background work. flock command documentation
Save the example as flock_demo.py and run python3 flock_demo.py. It creates the test file in a temporary directory and launches a new Python process for each competing lock attempt.
Verify closing versus deleting
The example ran on Linux x86-64 with Python 3.12.14 on November 18, 2026. The first competitor immediately failed to acquire the lock. After the owner closed the file, a competitor acquired it even though the path still existed. Finally, the example deleted the locked file and reopened the same path, checking that the new file could be locked while the original lock was still held. It also checked that the two open files had different inode numbers.
An unlinked file remains available through existing open descriptors. A new file created at the same path is a different file, allowing the processes to hold exclusive locks on different files. Keep a stable lock path while processes may be running or waiting to acquire it. Linux unlink(2) · Linux flock(2)
Check the batch environment
flock is a cooperative lock that requires participating processes to follow the same rule. An execution path that skips locking can still run concurrently. If separate containers create the same pathname in independent filesystems, check whether they actually lock the same file. NFS and SMB locking behavior can vary with clients, servers, and mount options. This example verifies process contention on a local temporary file. Actual schedulers and network filesystems were outside the test scope. Linux flock(2)