Ping Works: Setting Up Ubuntu 24.04 VXLAN Overlay Network for Two Hosts

Establishing a VXLAN overlay between two Ubuntu 24.04 hosts requires configuring the Linux kernel module and using ip link commands, as Netplan does not natively support VXLAN creation in its standard YAML syntax. This guide details the manual configuration steps to create a virtual network interface that encapsulates traffic over your existing physical connection.

Prerequisites and Kernel Module Loading

Before configuring the overlay, ensure the underlying infrastructure is ready. Both Host 1 (H1) and Host 2 (H2) must be running Ubuntu 24.04 LTS. The primary requirement is that the vxlan kernel module is loaded on both machines. You can verify this by running:

lsmod | grep vxlan

If the module is not listed, load it manually with:

sudo modprobe vxlan

To make this persistent across reboots, add vxlan to /etc/modules-load.d/vxlan.conf. Ensure that your physical network interfaces are up and that you have basic IP connectivity between the two hosts. If you are unsure how your server interfaces are currently managed, refer to our guide on How do Ubuntu-server network interfaces work? to understand the difference between legacy ifupdown and modern NetworkManager or Netplan backends.

Configuring Host 1 (H1)

On the first host, we will create a VXLAN interface named vxlan0. We assign it a Virtual Network Identifier (VNI) of 100. The critical parameter is remote, which points to the public or private IP address of Host 2. Execute the following command on H1. Replace H2_IP with the actual IP address of the second host.

sudo ip link add dev vxlan0 type vxlan id 100 remote H2_IP dstport 8472 dev eth0
sudo ip addr add 10.0.0.1/24 dev vxlan0
sudo ip link set vxlan0 up

Breakdown of the flags:

• id 100: The VNI. This must match the VNI configured on H2.
• remote H2_IP: The destination IP for encapsulated packets.
• dstport 8472: The standard UDP port for VXLAN traffic.
• dev eth0: The physical interface used for underlay routing.

After execution, verify the interface status with ip link show vxlan0. It should show state UP. Assign the IP 10.0.0.1/24 to this new virtual interface.

Configuring Host 2 (H2)

The process on the second host is symmetric but reversed regarding the remote IP. On H2, run the equivalent commands, replacing H1_IP with the IP address of Host 1.

sudo ip link add dev vxlan0 type vxlan id 100 remote H1_IP dstport 8472 dev eth0
sudo ip addr add 10.0.0.2/24 dev vxlan0
sudo ip link set vxlan0 up

It is crucial that the VNI (id 100) matches exactly what was set on H1. If the VNIs differ, the kernel will drop the encapsulated packets because it treats them as belonging to different logical networks. Also, ensure the subnet mask (/24) is consistent so that ARP resolution works correctly within the overlay.

Testing Connectivity

Once both interfaces are up, test the overlay by pinging from H1 to H2’s overlay IP:

ping 10.0.0.2

If the ping succeeds, the VXLAN tunnel is operational. If it fails, check the following:

1. Firewall Rules: Ensure UDP port 8472 is open on both hosts. Use sudo ufw allow 8472/udp if using UFW.
2. Routing: Verify that the underlay route to the remote host exists. Run ip route get H2_IP on H1 to confirm it routes through eth0.
3. MTU Issues: VXLAN adds 50 bytes of overhead. If your physical MTU is 1500, the effective MTU for the overlay is 1450. Consider setting mtu 1450 on the vxlan0 interface if you experience packet drops with larger payloads.

For more context on how VMs communicate through hosts, see How to Configure a Traffic Route from VM1 through Host to VM2, as the principles of underlay routing apply similarly here.

Making Configuration Persistent

Since these commands are run via iproute2, they will disappear after a reboot. To persist the configuration on Ubuntu 24.04 without relying on unsupported Netplan VXLAN blocks, you can use a systemd service or a network dispatcher script. Create a file at /etc/systemd/system/vxlan-setup.service:

[Unit]
Description=Setup VXLAN Overlay
After=network-pre.target
Before=network.target

[Service]
Type=oneshot
ExecStart=/bin/bash /usr/local/bin/setup-vxlan.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Then create the script /usr/local/bin/setup-vxlan.sh:

#!/bin/bash
# Wait for network to be ready
sleep 10

# Define variables
VNI=100
REMOTE_IP="H2_IP" # Update this per host
LOCAL_IP="10.0.0.1" # Update this per host
PHYS_DEV="eth0"

# Remove existing interface if present
ip link del vxlan0 2>/dev/null

# Create VXLAN
ip link add dev vxlan0 type vxlan id $VNI remote $REMOTE_IP dstport 8472 dev $PHYS_DEV

# Set IP and bring up
ip addr add $LOCAL_IP/24 dev vxlan0
ip link set vxlan0 up

Make the script executable with chmod +x /usr/local/bin/setup-vxlan.sh and enable the service with systemctl enable vxlan-setup.service. Note that you must update the REMOTE_IP and LOCAL_IP values in the script for each host individually.

Troubleshooting Common Issues

If you encounter issues where ping works but throughput is low, check for asymmetric routing. Ensure that return traffic from H2 also goes through the VXLAN interface. You can inspect the ARP table on H1 with ip neigh show dev vxlan0. If the entry for 10.0.0.2 is incomplete, try flushing the cache with ip neigh flush dev vxlan0 and pinging again. Additionally, verify that no other services are conflicting with the IP addresses assigned to the overlay. If you previously had a VM setup that used similar subnets, ensure there is no overlap. For those migrating from older Ubuntu versions, note that networking behaviors may differ slightly from Ubuntu 22.04 setups, as discussed in Ubuntu 22.04 VM on KVM/virt-manager Not Connecting to Internet.

Frequently Asked Questions

Why doesn't Netplan support VXLAN directly?

Netplan is primarily a renderer abstraction layer that generates configurations for NetworkManager or systemd-networkd. Neither backend has native, stable support for creating VXLAN interfaces via their primary configuration files in Ubuntu 24.04. Using iproute2 scripts provides a reliable workaround that bypasses these limitations.

What happens if the VNIs do not match?

If the VNI (Virtual Network Identifier) differs between the two hosts, the kernel will treat the incoming packets as belonging to a different logical network. Consequently, the packets will be dropped, and you will not be able to ping or communicate between the overlay interfaces, even if the physical network is perfectly connected.

Can I use a different UDP port than 8472?

Yes, you can specify any valid UDP port using the dstport flag. However, 8472 is the IANA-assigned default port for VXLAN. Using non-standard ports may cause issues with firewalls or network appliances that expect standard VXLAN traffic patterns, so sticking to 8472 is recommended unless required otherwise.

How do I debug VXLAN packet drops?

Use tcpdump -i eth0 udp port 8472 on both hosts to verify that encapsulated packets are being sent and received. If you see outgoing packets but no incoming ones, check your firewall rules (UFW or iptables) to ensure UDP 8472 is allowed inbound. Also, verify that the remote IP resolves correctly and is reachable via ICMP ping.

Does this setup work with IPv6?

The examples provided use IPv4. To use IPv6, replace the IPv4 addresses with IPv6 equivalents and ensure the dev parameter points to an interface capable of IPv6 routing. The logic remains identical, but you must configure the appropriate IPv6 prefixes and ensure your underlay network supports IPv6 routing to the remote host.