Understanding Why Standard Routes Fail
When you add a simple static route using commands likeip route add, you are telling the kernel how to reach a specific network. However, this does not dictate which local IP address should be used as the source for that traffic. If both NICs are on different subnets but the system attempts to send packets out the wrong interface, the return traffic will not find its way back. The failure is rarely due to physical connectivity but rather a logical mismatch in the kernel’s Forwarding Information Base (FIB). Without policy-based routing, the OS cannot distinguish between "go to Network A via NIC 1" and "go to Network B via NIC 2" if both destinations are reachable through a default gateway or if the subnet masks overlap incorrectly.
Configuring Policy-Based Routing
To successfully interconnect two discrete NICs, you must implement Policy-Based Routing (PBR). This involves creating separate routing tables for each NIC and binding them to the specific IP addresses assigned to those interfaces. First, identify the unique routing table ID you want to use for the second NIC, such as100. Create a new file or entry in /etc/iproute2/rt_tables to ensure this ID is recognized by the system. Then, populate this new table with the default gateway and subnet information specific to the second NIC. For example, you might add a default route in table 100 pointing to the gateway associated with the second NIC.
Next, you need to create a rule that directs traffic originating from the first NIC’s IP address to use the primary routing table, and traffic from the second NIC’s IP address to use table 100. Use the command ip rule add from [IP_OF_NIC_1] table main and ip rule add from [IP_OF_NIC_2] table 100. This ensures that any packet sent from the second NIC will look up its route in the secondary table, forcing it out through the correct physical interface. This method resolves the ambiguity that causes standard routing manipulations to fail.
Verifying Firewall and NAT Settings
Even with correct routing tables, traffic may still be blocked if the host firewall or Network Address Translation (NAT) settings are misconfigured. Many users forget that interconnecting NICs often requires masquerading if the NICs connect to different private networks that do not have overlapping subnets. Check your iptables or nftables rules to ensure that forwarding is enabled and that SNAT (Source NAT) is applied if necessary. If the two NICs are connecting to isolated segments, you may need to enable IP forwarding in the kernel by settingnet.ipv4.ip_forward=1 in /etc/sysctl.conf. Without this setting, the kernel will drop packets that appear to be forwarded between interfaces, regardless of how perfect your routing table is.
For more complex scenarios involving bandwidth aggregation or load balancing across multiple links, consider reviewing Can I Use Two NICs to Get Double the Bandwidth to a Switch? to understand the limitations of simple routing versus bonding protocols. Sometimes, the issue is not routing but rather that the switch or upstream device is not configured to handle multiple MAC addresses or VLAN tags correctly.
Troubleshooting Common Symptoms
If your configuration still fails, start by verifying the active rules withip rule show and ip route show table 100. Ensure that the source IP matches exactly what is assigned to the NIC; even a slight discrepancy can cause the rule to be ignored. Use tcpdump -i [NIC_NAME] on both interfaces to see if packets are actually leaving the correct card. If you see packets arriving on one NIC but never leaving the other, check the ARP tables to ensure that the next-hop gateway is resolvable. Additionally, verify that there are no conflicting static routes in the main table that have a longer prefix match than your policy rules.
In cases where you are trying to bridge these connections to another network segment, such as behind a router, How to setup a second router behind a different router without changing the operating mode provides insights into handling nested network configurations that might interfere with direct NIC-to-NIC routing.
Frequently Asked Questions
Why does my ping work but HTTP requests fail?
This usually indicates that ICMP is allowed by the firewall while TCP ports are blocked or that NAT is not configured for the higher-layer protocols. Ensure that your firewall rules explicitly permit established connections and that SNAT is applied if the destination network expects a translated source address.
Do I need to restart the network service after adding rules?
No, ip rule and ip route changes take effect immediately in the kernel memory. However, if you modified /etc/sysctl.conf or /etc/iproute2/rt_tables, you should reload the sysctl settings or reboot to ensure persistence across reboots.
What happens if two NICs are on the same subnet?
If both NICs share the same subnet, the kernel will struggle to determine which interface to use for local traffic, often leading to asymmetric routing. You should either place them on different subnets or use VLAN tagging to logically separate the traffic streams.
Can I use bonding instead of manual routing?
Bonding (LACP or balance-rr) combines NICs into a single logical interface, which simplifies routing but requires switch support and does not provide separate routing paths for different traffic types. Manual policy routing offers more granular control over which traffic goes where, whereas bonding aggregates bandwidth.
How do I make these routing rules persistent?
You must save the ip rule and ip route commands in a startup script or network configuration manager like Netplan or NetworkManager dispatcher scripts. Simply running them in the terminal will result in their loss after a system restart.