Troubleshooting Traffic Forwarding with Multiple Machines on a LAN

Traffic forwarding issues on a Linux LAN often stem from incorrect routing tables, disabled IP forwarding, or firewall rules blocking inter-host communication. By using tcpdump to capture packets and verifying kernel parameters, you can isolate whether the problem lies in the network stack configuration or physical connectivity, ensuring seamless backend service communication.

Verifying Kernel Forwarding Configuration

The most common reason for traffic failing to forward between machines on the same subnet is that the Linux kernel has IP forwarding disabled by default. While this is secure for standalone workstations, it breaks functionality when a machine acts as a bridge or router for other services. You must check the current state of the net.ipv4.ip_forward parameter to ensure it is enabled. Run the following command to view the current status:

sysctl net.ipv4.ip_forward

If the output is net.ipv4.ip_forward = 0, forwarding is disabled. To enable it temporarily for the current session, use:

sudo sysctl -w net.ipv4.ip_forward=1

For a permanent change that survives reboots, add or modify the line net.ipv4.ip_forward = 1 in your /etc/sysctl.conf file or create a drop-in file in /etc/sysctl.d/. After editing, apply the changes with sudo sysctl -p. This step is critical because even if your routing tables are correct, the kernel will drop any packet not destined for its own IP address if this flag is off.

Diagnosing with Tcpdump

Once the kernel settings are verified, the next step is to observe actual packet flow. tcpdump is an indispensable tool for seeing exactly what enters and leaves each network interface. When troubleshooting multiple machines, you need to determine if packets are arriving at the source but failing to leave, or if they are being dropped entirely by intermediate devices. On the machine receiving the forwarded traffic, run:

sudo tcpdump -i eth0 port 8080 -vvv

Replace eth0 with your specific interface name and 8080 with the relevant backend service port. If you see no output while sending requests from another machine, the traffic is not reaching this host. Check the source machine’s outgoing interface as well. If the source shows the packet leaving but the destination shows nothing, the issue likely lies in the middle layer, such as a switch misconfiguration or an upstream firewall. It is also helpful to filter by MAC addresses to ensure you are looking at the correct broadcast domain. Use -e to include link-layer headers in the output, which helps identify if ARP resolution is failing. Poor ARP caching can cause intermittent forwarding failures where packets are sent to the wrong hardware address.

Checking Firewall Rules

Even with forwarding enabled and correct routing, local firewalls like iptables or nftables may block inter-host traffic. Linux systems often have strict default policies that drop incoming connections unless explicitly allowed. You must verify that the FORWARD chain (if acting as a router) or INPUT/OUTPUT chains (if acting as endpoints) permit the traffic. Check your current iptables rules with:

sudo iptables -L -n -v

Look for rules that DROP or REJECT traffic on the ports used by your web-service backends. If you find restrictive rules, you may need to add ACCEPT rules for trusted subnets. For example:

sudo iptables -A INPUT -p tcp --dport 8080 -j ACCEPT

Modern distributions may use ufw or firewalld as front-ends. Ensure these tools are not overriding your manual configurations. Inconsistent firewall states across multiple machines can lead to asymmetric routing issues where one side accepts traffic but the other drops responses, causing connection timeouts.

Routing Table Analysis

Incorrect routing tables can direct traffic to the wrong gateway or interface. Use ip route show to display the active routing table. Each machine should have a direct route to the local subnet via its own interface, rather than pointing to a default gateway for internal traffic.

ip route show

Ensure that the metric for local routes is lower than that of external gateways. If a static route is misconfigured, traffic intended for a neighboring backend server might be routed out to the internet instead of staying on the LAN. Compare the routing tables of all involved machines to identify discrepancies. For more details on how the kernel decides between local delivery and forwarding, see our guide on [Understanding Packet Forwarding vs Local Delivery in Network Routing](/news/1172350/packet-forwarding-vs-local-delivery).

Frequently Asked Questions

Why does tcpdump show packets but the application doesn't receive them?

This usually indicates that the packet is reaching the network interface but is being blocked by the application firewall or not bound to the correct socket. Check if the service is listening on 0.0.0.0 or specifically on the LAN IP address using ss -tlnp.

How do I test connectivity without tcpdump?

You can use ping to verify basic layer 3 connectivity and telnet or nc to test layer 4 port availability. These tools provide quicker feedback than packet captures for simple reachability checks.

Can VLANs affect traffic forwarding between machines?

Yes, if machines are on different VLANs, they require a router or layer 3 switch to forward traffic. Direct bridging is not possible across VLAN boundaries without additional configuration like 802.1Q tagging on the interfaces.

What causes intermittent packet loss in LAN forwarding?

Intermittent loss is often due to duplex mismatches, faulty cables, or ARP cache poisoning. Verify that all switches and NICs are operating at full duplex and check for CRC errors in the interface statistics using ip -s link.

Is it safe to disable the firewall for testing?

Disabling the firewall can help isolate whether rules are the cause of the failure, but it exposes the system to risks. Always re-enable it immediately after testing and configure specific allow rules instead of leaving it completely open.