Parsing Mosquitto MQTT Broker Log Messages: Understanding 'New Connection' Messages

Mosquitto logs use the log_type directive to control output, not log_level. The standard format for a new connection is New client connected from IP:port as ID (c0, k60). To parse this reliably in Bash or Python, extract the client identifier from the fifth field after splitting by spaces, while noting that anonymous clients appear as "anonymous" rather than a UUID.

Configuring Log Output Types

To understand how to parse these messages, you must first ensure Mosquitto is configured to emit them. A common misconception is that changing the log level automatically changes the message structure. In reality, Mosquitto uses the log_type configuration option to select which categories of events are written to the log file or stdout. This setting accepts multiple values separated by commas. The most relevant types for connection monitoring are connection, disconnection, and error. If your logs are empty or missing connection events, verify your mosquitto.conf contains a line such as:

log_type connection
log_type disconnection
log_type error

Without explicitly enabling connection, the broker will not write the detailed handshake information required for parsing. You can also direct these logs to specific destinations using log_dest file /var/log/mosquitto/mosquitto.log or log_dest stdout. Understanding this distinction prevents the futile attempt to filter logs that were never generated in the first place due to incorrect type selection.

The Standard Connection Log Format

When a client successfully connects, Mosquitto generates a specific string pattern. Recognizing this pattern is crucial for accurate parsing. The default format when log_format standard is active looks like this:

New client connected from 192.168.1.50:54321 as ClientID123 (c0, k60).

Breaking down this structure reveals the data points available for extraction:

• Prefix: The literal string "New client connected from".
• Address: The IPv4 or IPv6 address followed by a colon and the ephemeral port number.
• Client ID: The unique identifier assigned to the client session. This appears immediately after the word "as".
• Flags: Enclosed in parentheses, c0 indicates clean session state, and k60 represents the keep-alive interval in seconds.

It is critical to note that if the client does not provide a Client ID during the CONNECT packet, Mosquitto assigns one internally but logs it as "anonymous". Therefore, any parser expecting a hexadecimal string or UUID in the client ID position will fail on unconfigured IoT devices. Always account for the possibility of the string "anonymous" appearing in the expected ID slot.

Parsing with Command Line Tools

For system administrators needing to monitor connections in real-time, command-line tools offer efficient solutions. While previous drafts suggested using awk with fixed field indices, the reliability depends on consistent spacing. The standard log format uses single spaces between fields, making awk a viable option if applied correctly. To extract just the Client ID from a stream of logs, you can use:

grep "New client connected" mosquitto.log | awk '{print $5}'

This command filters for connection events and prints the fifth whitespace-delimited field. In the example New client connected from 192.168.1.50:54321 as ClientID123..., the fields are:

1. New
2. client
3. connected
4. from
5. 192.168.1.50:54321
6. as
7. ClientID123

Wait, the previous analysis shows that field 5 is the IP:Port, not the ID. The Client ID is actually the seventh field if we count strictly by space separation in the standard format. Let us re-evaluate.

New(1) client(2) connected(3) from(4) 192.168.1.50:54321(5) as(6) ClientID123(7)

Thus, the correct awk command to get the Client ID is:

grep "New client connected" mosquitto.log | awk '{print $7}'

If you need both the IP and the ID, you can capture them together:

grep "New client connected" mosquitto.log | awk '{print $5, $7}'

This approach assumes the standard log format. If you have switched to JSON logging via log_format json, this text-based parsing will fail entirely, requiring a different toolset.

Handling Edge Cases and Anonymous Clients

Parsing logic must be robust enough to handle non-standard client behaviors. As mentioned, anonymous clients are logged as "anonymous" instead of a unique ID. If your application relies on tracking specific device IDs, you might want to filter these out or flag them for security review. Additionally, consider the impact of IPv6 addresses. An IPv6 address like ::1 does not contain colons that might confuse simple parsers looking for the last colon as a separator between IP and Port. However, since the format always includes the port after a colon (e.g., [::1]:54321 for some implementations or just ::1:54321), relying on the positional index ($5 for IP, $7 for ID) remains safer than regex-based splitting on delimiters. For more complex environments where MQTT integration is challenging, see our guide on Solving Difficult MQTT Connection Issues in Node-RED on Raspberry Pi. This resource covers broader connectivity problems that may manifest as irregular log entries.

Alternative: JSON Logging for Structured Data

If you are building a scalable monitoring system, parsing plain text logs is fragile. Mosquitto supports JSON logging, which structures each event into a machine-readable object. Enable this by adding log_format json to your configuration. A JSON-formatted connection event looks like this:

{"timestamp":"2023-10-27T10:00:00Z","type":"connection","clientid":"ClientID123","ipaddress":"192.168.1.50","port":54321}

Using tools like jq, you can extract data without worrying about field positions:

cat mosquitto.log | jq -r 'select(.type=="connection") | .clientid'

This method is significantly more reliable for production environments. It handles special characters in client IDs gracefully and provides explicit keys for IP and port, eliminating ambiguity. For scripts that need to run with elevated privileges to access these logs securely, refer to Run Programs with Elevated Privileges: A Comprehensive Guide.

Frequently Asked Questions

Why are my connection logs missing?

Check your mosquitto.conf file to ensure log_type connection is explicitly enabled. By default, some configurations only enable errors, hiding normal connection activity.

How do I parse IPv6 addresses correctly?

Use positional field extraction (like awk '{print $5}') rather than splitting on colons. IPv6 addresses contain multiple colons, which breaks delimiter-based parsers.

What does "anonymous" mean in the logs?

It indicates the connecting client did not provide a Client ID in its CONNECT packet. Mosquitto assigns a temporary internal ID but logs it as "anonymous" for privacy and simplicity.

Can I use grep to find disconnected clients?

Yes, but you must also enable log_type disconnection in your config. The log message will start with "Client disconnected" instead of "New client connected".

Is JSON logging better than standard text logs?

JSON is superior for automated parsing and integration with monitoring systems. It avoids issues with variable spacing and special characters found in text-based formats.