Print Chosen Columns of lsof Output with Unique-c PID and Sort on MacOS using awk, sort, and uniq

To print chosen columns from lsof output while filtering for unique PIDs and sorting the results on macOS, pipe the command through awk to extract fields, sort -k2,2n to order by PID numerically, and uniq -d or awk '!seen[$0]++' to handle uniqueness. This approach leverages standard Unix utilities available in the macOS terminal to parse complex process lists efficiently without external dependencies.

Understanding lsof Output Structure

The lsof (list open files) command provides a detailed view of all files currently open by processes on your system. On macOS, this output can be extensive, often containing hundreds or thousands of lines depending on the number of active applications and system services. The default output format includes several columns: COMMAND, PID, USER, FD, TYPE, DEVICE, SIZE/OFF, NODE, and NAME. When you need to analyze specific aspects of this data, such as identifying which processes are holding specific resources, you rarely need the entire line. Instead, you typically want to isolate specific fields like the PID and the file name. However, raw lsof output is not sorted, and duplicate entries may appear if multiple instances of a process are running or if the same file is opened by different threads. Understanding this structure is crucial before applying any filtering or sorting logic.

Extracting Specific Columns with Awk

The first step in refining the lsof output is to select only the columns you care about. The awk utility is perfect for this task because it treats each line as a record and each whitespace-separated value as a field. For instance, if you want to see the PID (field 2) and the NAME (field 9), you can use the following command:

lsof | awk '{print $2, $9}'

This command pipes the output of lsof directly into awk. Inside awk, the $2 refers to the second column (PID) and $9 refers to the ninth column (NAME). You can adjust these indices based on your specific needs. If you also need the USER (field 3), you would add $3 to the print statement. Using awk ensures that you strip away unnecessary noise from the output, making subsequent processing steps faster and more accurate.

Sorting by PID Numerically

Once you have extracted the relevant columns, the next logical step is often to sort the data. By default, the sort command sorts alphabetically. This means that PID 10 will appear after PID 2 but before PID 3, which is usually not what you want when dealing with process IDs. To sort numerically, you must specify the key and the numeric flag. Assuming you are working with the two-column output from the previous step (PID and NAME), you can sort by the first column (PID) using:

lsof | awk '{print $2, $9}' | sort -k1,1n

Here, -k1,1n tells sort to use the first field as the key and to interpret it as a number (n). This ensures that PIDs are ordered correctly from lowest to highest. If you prefer descending order, you can add the r flag: sort -k1,1nr. Proper sorting is essential for readability and for ensuring that duplicate detection works consistently, as uniq only removes adjacent duplicates.

Handling Unique Values and Deduplication

After sorting, you might want to eliminate duplicate entries. In the context of lsof, duplicates might arise if the same process has opened the same file multiple times, or if you are looking for unique PIDs regardless of the file they have open. If you want to keep only unique combinations of PID and NAME, you can append uniq to your pipeline:

lsof | awk '{print $2, $9}' | sort -k1,1n | uniq

However, uniq only removes consecutive duplicate lines. This is why sorting is mandatory before using uniq. If you want to find PIDs that appear more than once (i.e., processes with multiple open files), you can use uniq -d:

lsof | awk '{print $2}' | sort -n | uniq -d

Alternatively, for more complex deduplication logic where you want to keep the first occurrence of each PID but ignore subsequent ones, awk can handle this internally without needing sort or uniq:

lsof | awk '!seen[$2]++ {print $2, $9}'

This awk one-liner uses an associative array seen to track PIDs. It prints the line only if the PID has not been seen before. This method is efficient and does not require sorting, though the output will not be ordered by PID unless you sort afterward.

Integrating with System Monitoring Tools

These techniques are particularly useful when troubleshooting resource leaks or monitoring system stability. For example, if you suspect a process is leaking file descriptors, you can monitor the count of open files per PID over time. Combining these shell commands with cron jobs or simple scripts allows for automated reporting. If you are interested in deeper process analysis, you might also look into understanding SSHd process output on macOS to see how network daemons behave under load. Additionally, for those managing containerized environments, getting started with Minikube on macOS offers insights into how virtualized processes interact with the host system's file descriptors. These related topics provide broader context for system administration tasks on macOS.

Frequently Asked Questions

Why does my lsof output include header lines?

The default lsof output includes a header row describing each column. To exclude this, you can pipe the output through tail -n +2 after lsof but before awk, or simply tell awk to skip the first line by adding NR>1 to the condition, like awk 'NR>1 {print $2, $9}'.

Can I filter lsof output for a specific user?

Yes, you can add a -u username flag to the lsof command itself, such as lsof -u $USER. Alternatively, you can filter within awk by checking the third field: awk '$3 == "username" {print $2, $9}'.

What if I want to sort by the filename instead of PID?

You can modify the sort command to target the second field. If your awk command outputs PID then NAME, use sort -k2,2 to sort alphabetically by the filename. Add r for reverse alphabetical order.

How do I handle special characters in filenames during parsing?

lsof output can sometimes contain spaces or special characters in filenames, which may break awk field splitting. In such cases, consider using lsof -F n (which outputs null-separated fields) or parsing with more robust tools like perl or python if the data becomes too complex for simple shell pipelines.

Is there a way to limit the number of results shown?

Yes, you can append head -n 10 to the end of your pipeline to display only the first ten lines of the processed output. This is useful for quick previews when testing your command syntax before running it against the full system state.