jdupes uses the --link flag to replace duplicate files with hardlinks, preserving one copy and pointing all others to the same inode. To control which file is kept as the primary reference, you must use the --keep-newer, --keep-older, or --keep-first options explicitly. Without these flags, jdupes defaults to keeping the first file encountered in its scan order, which can lead to unexpected retention of temporary or system files if not managed carefully.
Understanding Hardlinks and Deduplication
Before executing any deduplication commands, it is crucial to understand what a hardlink actually does. A hardlink is not a shortcut or a symbolic link; it is an additional directory entry that points to the same underlying data blocks on the disk. When you create a hardlink, you are essentially giving the same file two different names within the filesystem structure. The operating system tracks how many hardlinks point to a specific inode. The data remains on the disk only as long as at least one hardlink exists. If you delete all hardlinks for a particular set of duplicates, the data is permanently removed. This behavior is fundamentally different from creating copies. When you use standard copy commands, each file occupies its own separate space on the storage medium. With hardlinks, multiple filenames share the exact same physical location. This means that modifying one "duplicate" will instantly reflect in all other linked files because they are, technically, the same file. This property makes hardlinks incredibly efficient for saving space, especially when dealing with large media libraries or backup directories where identical files exist in multiple locations. For more context on how the operating system tracks these links, you might find Understanding the Hardlinks Field of a Directory helpful.The Standard Deduplication Command
The core command used to perform deduplication via hardlinks in jdupes is--link. This is the standard and most commonly used mode for reducing disk usage. When you run jdupes with this flag, the tool scans the specified directories, identifies files with identical content, and then replaces the duplicates with hardlinks. It is important to note that there is no such thing as a --harder flag in jdupes. Any guide suggesting otherwise is incorrect. The correct syntax involves specifying the target directories followed by the --link option.
For example, a basic command might look like this:
jdupes --link /path/to/source/dir--link is inherently destructive to the duplicate files. It removes the duplicate entries and creates hardlinks in their place. Always ensure you have backups before running this command, as the operation cannot be easily undone. The tool will prompt you for confirmation before making changes, but relying on prompts alone is risky in automated scripts.
Controlling Which File Is Preserved
One of the most critical aspects of using jdupes is deciding which file remains after deduplication. By default, jdupes keeps the first file it encounters in its internal list of duplicates. This order is determined by how the filesystem returns directory entries, which is often arbitrary and not based on creation time or alphabetical order. This unpredictability can cause issues if you want to preserve newer versions of documents or older versions of archived data. To gain precise control, you should use specific flags that dictate the retention policy. The--keep-newer flag ensures that the most recently modified file is preserved, while all older duplicates are replaced with hardlinks to it. This is ideal for document management where you want to keep the latest edits. Conversely, --keep-older preserves the oldest file, which might be useful for maintaining historical archives without deleting early versions. Another option is --keep-first, which explicitly enforces the default behavior of keeping the first file found, making your intent clear to anyone reading the script.
You can combine these flags with other options to refine the process. For instance, you might want to exclude certain types of files from being linked. Using --exclude allows you to specify patterns for files that should never be touched, regardless of their duplication status. This prevents accidental hardlinking of system files or configuration files that might behave unexpectedly if shared across multiple paths.
Verification and Safety Checks
Before committing to a full deduplication, it is wise to run jdupes in report mode. Using the--print flag allows you to see exactly which files would be affected without making any changes. This output lists groups of duplicate files, showing you which ones would be kept and which would be replaced. Reviewing this list helps you verify that your retention policies are working as expected. You can also use --quiet to suppress verbose output during the actual linking phase, keeping your terminal clean.
Additionally, consider the implications of hardlinks on file permissions and ownership. Since hardlinks share the same inode, they also share the same permissions and ownership attributes. If you have files with different permission requirements that happen to have identical content, hardlinking them will force them to share the same permissions. This can lead to security issues if a file intended to be private becomes accessible due to its hardlinked counterpart. Always audit your file permissions after running jdupes to ensure no unintended access has been granted.
Common Pitfalls and Best Practices
A common mistake users make is assuming that hardlinks work across different filesystems or partitions. Hardlinks rely on the underlying inode structure, which is local to a single filesystem. If you attempt to hardlink a file from an external USB drive to an internal SSD, jdupes will fail because the inodes are not compatible across different mount points. In such cases, you must use symbolic links or copy the data instead. Understanding these limitations prevents wasted time and potential data loss. Another pitfall involves symlinks. If your directory contains symbolic links that point to duplicate files, jdupes may follow these links and create hardlinks to the target of the symlink. This can result in complex chains of links that are difficult to manage. It is generally recommended to use the--no-symlinks flag if you want to avoid following symbolic links during the scan. This ensures that only regular files are considered for deduplication.
Finally, always test your setup on a small subset of files before applying it to your entire library. Create a temporary directory with known duplicates and run your preferred jdupes command. Verify that the correct files are preserved and that the space savings match your expectations. This practice builds confidence in your workflow and helps you catch any configuration errors before they impact your valuable data. For related insights on managing file histories, you may also want to read How to Clear Recent Files History in Linux without Deleting Actual Files? to understand how metadata interacts with file structures.