Using mlocate to search on multiple systems in parallel

The mlocate command itself cannot search multiple systems or remote databases in parallel because it only reads a single local database file. To achieve parallel searching across multiple servers, you must combine the standard locate utility with SSH and a parallel execution tool like GNU Parallel or xargs. This approach allows you to query each remote system’s local index simultaneously, aggregating the results efficiently without overloading any single network connection.

Understanding the Limitations of mlocate

It is crucial to understand that mlocate, which is often symlinked to locate on many Linux distributions, operates strictly on local data. The core binary does not have built-in networking capabilities, nor does it support flags like -d to specify multiple database files or remote hosts. Attempting to use options intended for other tools will result in errors or unexpected behavior. The locate command works by querying a pre-built database, typically located at /var/lib/mlocate/mlocate.db. This database is updated periodically by the updatedb service. Because this database resides on the local filesystem, any search performed by locate is confined to the machine running the command. Therefore, the strategy for multi-system searches must shift from modifying locate to orchestrating multiple instances of it across different hosts. For more information on efficient text processing, see Combining Search for Multiple Strings and Printing Rows After a Hit in Grep.

Preparing Remote Systems

Before implementing parallel searches, ensure that all target systems are properly configured for secure, passwordless access. This setup eliminates interactive prompts that would block automated scripts. You should generate an SSH key pair on your local machine if you do not already have one.

1. Generate an SSH key using ssh-keygen -t ed25519.
2. Distribute the public key to all remote hosts using ssh-copy-id user@remote-host.
3. Verify connectivity by running ssh user@remote-host 'echo Connected'.

Ensure that updatedb has run recently on the remote systems so their databases are current. Stale data can lead to misleading search results. If the remote systems are heavily loaded, consider scheduling updates during off-peak hours to maintain performance.

Using GNU Parallel for Execution

GNU Parallel is a powerful shell tool designed to execute jobs in parallel using one or more computers. It is ideal for distributing locate commands across multiple servers. If it is not installed, you can install it via your package manager, such as sudo apt install parallel or sudo yum install parallel. The basic syntax involves defining a list of hosts and passing them to parallel. Each host runs the specified command independently. Below is a robust script structure for this task.

# Define the list of remote hosts
HOSTS="server1.example.com server2.example.com server3.example.com"

# Define the search pattern
PATTERN="config.yaml"

# Run locate in parallel on each host
echo "$HOSTS" | parallel ssh {} "locate $PATTERN"

This command sends the locate command to each host in the list. The output from each host is printed sequentially as it completes. While simple, this method provides significant speed improvements over sequential SSH loops. For users interested in managing terminal sessions while these processes run, tmux: Searching Certain Strings in Scrollback Unreliable offers insights into session management.

Advanced Filtering and Output Handling

Raw output from parallel searches can be difficult to parse because it lacks context about which host returned which result. To improve usability, you can prepend the hostname to each line of output. This is achieved by modifying the SSH command slightly.

echo "$HOSTS" | parallel ssh {} "echo \$(hostname); locate $PATTERN"

This modification ensures that every line of output is preceded by the name of the server where the file was found. You can pipe this output into grep or awk for further filtering. For example, to count occurrences per host:

echo "$HOSTS" | parallel ssh {} "echo \$(hostname); locate $PATTERN" | awk '{print $2}' | sort | uniq -c

Note that the awk command here assumes the hostname is the first field and the file path starts after it. Adjust the field indices based on your specific output format.

Alternative Approaches with xargs

If GNU Parallel is unavailable, xargs can serve as a fallback. However, xargs does not handle job control as gracefully as parallel, especially when dealing with large numbers of hosts or varying response times.

echo "$HOSTS" | xargs -I {} ssh {} "locate $PATTERN"

This command executes the SSH session for each host one by one unless combined with other mechanisms. It is less efficient than GNU Parallel but requires no additional installation beyond standard Unix utilities. Be cautious with special characters in hostnames or patterns when using xargs.

Performance Considerations

Running parallel searches increases network traffic and CPU usage on remote systems. Monitor these resources to avoid impacting critical services. Use the -j flag in GNU Parallel to limit the number of concurrent jobs. For example, -j 4 restricts execution to four simultaneous connections. Additionally, consider the size of the remote databases. Large databases may take longer to query, leading to bottlenecks. Regularly optimizing updatedb configurations on remote hosts can mitigate this issue.

Frequently Asked Questions

Can I search a remote database directly without SSH?

No, locate does not support direct remote database access. You must use SSH to execute the command on the remote machine where the database resides.

How do I handle hosts that are offline?

Add error handling to your script. GNU Parallel continues executing remaining jobs even if one fails. You can log failures by redirecting stderr to a separate file for later review.

Is there a risk of overwhelming remote servers?

Yes, especially if many hosts are queried simultaneously. Use the -j option in GNU Parallel to throttle concurrency and reduce load on remote systems.

Can I search for regular expressions with this method?

Standard locate uses glob patterns, not full regular expressions. Use find instead if you need regex support, though it will be significantly slower than locate.

How often should updatedb run on remote systems?

Run it frequently enough to keep data fresh, but not so often that it impacts performance. Daily updates are usually sufficient for most environments.