ICMP Type 3 Code 4 Demo: Handling Traffic with Higher MTUs

An ICMP Type 3 Code 4 message indicates that a packet was fragmented but the "Don't Fragment" (DF) flag was set, preventing the router from splitting it further. This error occurs when a sender transmits packets larger than the path's Maximum Transmission Unit (MTU). To resolve this, you must identify the correct MTU using diagnostic tools and adjust your network interface or application settings to match the bottleneck.

The Mechanics of Path MTU Discovery

When two Linux instances communicate over a network, such as within an Amazon Web Services (AWS) environment, they rely on the Internet Control Message Protocol (ICMP) to manage packet sizes. The process begins with Path MTU Discovery (PMTUD), where the source host attempts to send packets with the DF flag enabled. Routers along the path check if the packet size exceeds their outgoing interface MTU. If it does, and fragmentation is blocked by the DF flag, the router drops the packet and sends back an ICMP Type 3 Code 4 response. This response contains crucial information: the IP address of the router that dropped the packet and the maximum transmission unit size supported by that specific hop. Without this feedback, the sender would either flood the network with oversized packets or unnecessarily reduce its payload size, wasting bandwidth. Understanding this interaction is vital for maintaining efficient data transfer between cloud instances. For more context on how traffic moves through networks, you might find Understanding Packet Forwarding vs Local Delivery in Network Routing helpful, as it clarifies the distinction between transit and final delivery.

Reproducing the Error in AWS

To demonstrate this issue, we use two Linux instances in the same Virtual Private Cloud (VPC). First, determine the default MTU of your interfaces, which is typically 9001 bytes for AWS eni interfaces supporting jumbo frames, or 1500 bytes for standard Ethernet. We will simulate a scenario where one instance tries to send a packet larger than the receiving path can handle without fragmentation. Use the ping command with the -M do flag to enforce the DF bit. For example, attempting to ping the remote instance with a large payload:

ping -M do -s 1500 <remote_instance_ip>

If the path includes a device with a lower MTU limit (such as a NAT gateway or a security appliance configured with stricter limits), the ping will fail. You will not see a timeout; instead, you will receive an immediate error message indicating that the operation requires fragmentation but the DF flag is set. This is the practical manifestation of the ICMP Type 3 Code 4 error.

Detecting the Bottleneck

When the connection fails, the next step is to isolate the exact point of failure. While the ICMP message provides the MTU limit, it does not always clearly indicate which intermediate hop caused the drop if multiple routers are involved. A systematic approach involves reducing the packet size incrementally until the communication succeeds. Start with a payload size slightly above the suspected bottleneck. If the ping fails, reduce the size by 10-20 bytes and try again. Continue this binary search or linear reduction until you find the largest packet size that passes through successfully. This value represents the Path MTU. You can also use traceroute combined with varying packet sizes to observe where the silence occurs. Note that some firewalls block ICMP messages entirely, which can lead to black holes where no error is returned. In such cases, PMTUD fails silently, leading to TCP timeouts rather than explicit ICMP errors.

Configuring the Fix

Once you have identified the optimal MTU, you must configure your Linux instances to adhere to this limit. There are two primary methods: adjusting the network interface MTU or configuring the application to respect the path limit. For network-level adjustments, use the ip command to change the MTU of the interface:

sudo ip link set dev eth0 mtu 1450

Replace eth0 with your actual interface name and 1450 with the discovered safe MTU. This change persists only until the next reboot unless added to your network configuration files (e.g., /etc/network/interfaces or Netplan configurations). Alternatively, if you are running applications like web servers or database clusters, configure them to use smaller TCP segments. For TCP connections, the effective MTU is often reduced by headers (IP + TCP + ICMP overhead). A common safe MTU for IPv4 over standard internet links is 1460 bytes for payload, accounting for 40 bytes of header overhead. If you are dealing with complex routing scenarios involving multiple machines, reviewing Troubleshooting Traffic Forwarding with Multiple Machines on a LAN can provide additional insights into how different nodes handle these constraints.

Verifying the Solution

After applying the MTU changes, verify the connectivity using the same ping command with the -M do flag and the newly calculated payload size. The ping should succeed consistently. Additionally, monitor the connection using tcpdump to ensure that no ICMP errors are being generated during high-throughput transfers.

tcpdump -i eth0 icmp

If you see ICMP Type 3 Code 4 messages disappearing, your configuration is correct. Remember that MTU settings are per-interface and per-route. If you have multiple routes or bonding interfaces, ensure all relevant paths are adjusted. In AWS, remember that VPC peering or Transit Gateway attachments may have their own MTU constraints that differ from the instance level. Always test end-to-end performance after making changes to ensure that latency has not increased due to excessive segmentation.

Frequently Asked Questions

Why does my connection time out instead of showing an ICMP error?

Some firewalls or security groups are configured to drop ICMP messages entirely to prevent reconnaissance. When these blocks are in place, the sender never receives the Type 3 Code 4 reply, causing TCP sessions to hang until they timeout. You can detect this by checking if other protocols return errors while TCP stalls.

Can I increase the MTU beyond 1500 bytes in AWS?

Yes, AWS supports jumbo frames up to 9001 bytes for most instance types and ENI configurations. However, both the sending and receiving instances must support jumbo frames, and any intermediate networking equipment must also allow larger packets. If any hop in the path restricts the size, the effective MTU drops to the lowest common denominator.

What is the difference between ICMP Type 3 Code 4 and Code 13?

Type 3 Code 4 specifically means "Fragmentation Needed and Don't Fragment was Set." Code 13 refers to "Administratively Prohibited," which indicates that a firewall or access control list blocked the packet. Code 4 is a technical limitation of path capacity, while Code 13 is a policy-based rejection.

How do I check the current MTU of a remote server?

You cannot directly query a remote server's interface MTU via ICMP. Instead, you must perform Path MTU Discovery by sending progressively smaller packets with the DF flag set until one succeeds. The size of the largest successful packet minus header overhead gives you the path MTU.

Does changing the MTU affect TCP throughput?

Setting the MTU too low increases header overhead, reducing effective throughput. Setting it too high causes fragmentation issues if the path cannot handle it, leading to packet loss and retransmissions. Finding the optimal MTU ensures that each packet carries maximum useful data without requiring fragmentation, thus maximizing efficiency.