When an IPSec tunnel is established but tools like nmap and ping fail, the issue usually stems from asymmetric routing, missing return routes on the remote peer, or firewall rules blocking ICMP and TCP SYN packets. Verify that traffic is actually traversing the tunnel interface by checking packet counters, then ensure your local routing table directs specific subnets through the VPN gateway rather than the default internet route.
Diagnosing Traffic Flow and Asymmetric Routing
The most common reason for a successful IPSec handshake followed by silent packet loss is asymmetric routing. This occurs when outbound packets travel through the IPSec tunnel, but return packets bypass it and go directly to the internet via the default gateway. Because the source IP of the returning packet does not match the expected path, the Ubuntu VM discards it as invalid. To confirm this behavior, you must inspect the kernel routing table to see how different destination networks are handled. Use theip route show command to display your current routing configuration. Look for entries that specify the dev (device) associated with your IPSec connection. Typically, strongSwan or LibreSwan creates a virtual interface such as tun0 or ipsec0, though the exact name depends on your software version and configuration. If you see a default route pointing to your physical network interface (like eth0) without any specific rules for the remote subnet, your system is likely ignoring the tunnel for general traffic.
You should also verify that the tunnel interface is actually receiving traffic. Use ip -s link to view statistics for your network interfaces. Check the RX (receive) and TX (transmit) byte counters for the tunnel device. If these numbers remain static while you attempt to ping a remote host, the packets are never reaching the kernel’s network stack over the tunnel. This indicates either a misconfiguration in the IPSec policy or that the remote endpoint is dropping the packets before they enter the tunnel.
For a deeper look into whether packets are being encrypted, enable verbose logging in your IPSec daemon. In strongSwan, this often involves adjusting the charon.conf settings to increase log verbosity for the IKE and ESP protocols. Restart the service with systemctl restart strongswan and monitor the logs using journalctl -u strongswan -f. You should see entries indicating that packets are being encapsulated and sent out. If you see errors about "no proposal chosen" or "authentication failed," the tunnel parameters themselves are mismatched, which would prevent data transfer even if the control channel appears up.
Checking Local Firewall Rules
Another frequent culprit is the local firewall on the Ubuntu VM. Tools likenmap rely on TCP SYN packets, while ping uses ICMP Echo Requests. If your firewall is configured to drop or reject these protocols, the tunnel will appear established, but no diagnostic tool will work. Check your active firewall rules using iptables -L -n -v or ufw status verbose. Ensure that there are no explicit DROP or REJECT rules targeting the tunnel interface or the remote subnet.
If you are using ufw (Uncomplicated Firewall), remember that it may block incoming connections by default. However, since you are initiating the connection from the VM, outgoing traffic should generally be allowed. The critical check is whether the firewall is interfering with the return traffic. Some configurations mistakenly apply strict rules to the loopback or tunnel interfaces, preventing the kernel from accepting the decrypted packets. Temporarily disabling the firewall with sudo ufw disable can help isolate whether the firewall is the cause. If connectivity returns immediately after disabling it, you need to add specific allow rules for the remote subnet through the tunnel interface.
Verifying Remote Peer Configuration
Troubleshooting network issues requires looking at both ends of the connection. Even if your Ubuntu VM is correctly configured, the remote server might not have the necessary routes to send traffic back through the IPSec tunnel. If the remote peer sends replies via its own default gateway, those packets will arrive at your VM with a source IP that belongs to the remote subnet. Your Linux kernel will try to route this reply based on the destination IP (which is your VM's IP), potentially sending it out the wrong interface if the routing table is not precise. Ensure that the remote peer has a static route pointing to your public IP address via the IPSec tunnel. On many Linux-based remote peers, this is configured in/etc/ipsec.conf or via ip route add commands. Without this reverse route, the conversation breaks down immediately after the initial handshake. Additionally, verify that the remote peer’s firewall allows inbound traffic from your VM’s public IP address on the required ports. A blocked port on the remote side will result in timeouts that mimic tunnel failures.
Adjusting MTU Settings
IPSec adds overhead to each packet due to encryption headers and encapsulation. Standard Ethernet frames have a Maximum Transmission Unit (MTU) of 1500 bytes. When you add IPSec headers, the total size exceeds this limit, causing fragmentation. If fragmentation is blocked by intermediate firewalls or misconfigured routers, large packets are dropped entirely. This often manifests as small pings working but larger scans failing, or vice versa. Check the MTU setting on your tunnel interface usingip link show dev tun0 (replacing tun0 with your actual interface name). If it is set to 1500, try reducing it to 1400 or 1380 to accommodate the IPSec overhead. You can test this dynamically using ping -M do -s 1472 . If this fails, reduce the payload size until it succeeds, then adjust the interface MTU accordingly. This step resolves issues where the tunnel is up but data transfer is unreliable or incomplete.
Finalizing the Fix
Once you have verified routing, firewall rules, and MTU settings, restart the IPSec service to apply any changes cleanly. Usesudo systemctl restart strongswan or sudo ipsec restart depending on your installation. After the restart, re-test connectivity with ping and nmap. If the issue persists, consider capturing traffic on both ends using tcpdump to see exactly where packets are disappearing. Compare the captured packets to ensure that encryption and decryption are occurring symmetrically. For more complex scenarios involving multiple network interfaces, refer to guides on routing WAN traffic through IPSEC tunnels to ensure your policies are applied correctly to all relevant subnets.
Frequently Asked Questions
Why does ping work but nmap fails?
This typically indicates an MTU issue or a firewall rule blocking specific TCP ports. Ping uses small ICMP packets that fit within standard MTU limits, while nmap sends larger TCP SYN packets. If fragmentation is blocked, the larger packets are dropped. Check your MTU settings and ensure your firewall allows the TCP ports you are scanning.How do I check if my IPSec tunnel is actually passing data?
Use theip -s link show dev [interface_name] command to view byte counters. If the RX and TX counters increase when you generate traffic, the tunnel is passing data. If they remain static, the packets are not reaching the tunnel interface, suggesting a routing or policy error.
What causes asymmetric routing in IPSec setups?
Asymmetric routing happens when outbound traffic goes through the tunnel but return traffic takes the default internet route. This occurs if the remote peer lacks a specific route back to your private subnet via the tunnel. Fix this by adding a static route on the remote peer pointing to your private network through the IPSec gateway.Can a firewall on the Ubuntu VM block IPSec traffic?
Yes, if the firewall drops ICMP or TCP SYN packets before they reach the network stack. While the IPSec control channel (UDP 500/4500) might pass, application-layer traffic could be blocked. Check youriptables or ufw rules to ensure they are not rejecting traffic destined for the remote subnet over the tunnel interface.