Configuring VXLAN.

VXLAN (Virtual Extensible LAN) carries Ethernet frames inside UDP packets, so hosts on different IP networks can share one layer 2 segment. Each host runs a VXLAN Tunnel Endpoint (VTEP) that wraps outgoing frames in UDP port 4789 and unwraps incoming ones. To the machines on the overlay, servers in three different datacenters look like they're plugged into the same switch.

That solves a specific problem: you want a private network across servers that don't share one. The usual cases are hosts in different clouds, a colo rack plus cloud VMs, or a cluster that outgrew a single provider's VPC. VXLAN gives them a common subnet without asking either provider for anything. What it doesn't give you is encryption, which matters as soon as that UDP traffic crosses the public internet.

Private Networking by Default on Cycle

A hand-built overlay works for a few servers. It gets expensive as the fleet grows, because every host needs VTEP config, forwarding entries for every peer, firewall rules, MTU tuning and a separate encryption layer on top.

If that's not something you want to manage yourself, Cycle handles it in the control plane. Every environment gets its own private network, and that network spans servers and providers (AWS, GCP, bare metal and others) as one flat network. When a server joins a cluster, Cycle maps latency, reachability and topology to its neighbors and routes traffic automatically. Nodes with a private link use it. Nodes without one get an IPsec-encrypted link over the public internet, with keys Cycle generates and rotates.

You don't write VTEP configs or forwarding tables, and you don't bolt encryption on afterward. See how Cycle networking works.

If you run your own servers, or want to understand what the control plane is doing for you, the rest of this page walks through configuring VXLAN by hand.

How VXLAN works

VXLAN is an encapsulation format, defined in RFC 7348. For background on overlays in general, see the VXLAN overview. The moving parts on Linux:

A VTEP has to know where to send a frame for a given MAC. There are three ways to answer that: a multicast group, a control plane like BGP EVPN, or static FDB entries you write yourself. Most clouds don't route multicast between hosts, so static entries are the common choice for small deployments, and they're what this page uses.

Encapsulation overhead

Each frame picks up new headers on the way out:

On a standard 1500-byte underlay, the VXLAN interface MTU must be 1450. With an IPv6 underlay the outer header is 40 bytes, so the overhead is 70 and the MTU drops to 1430.

Before you start

The examples use three servers with public IPs and one overlay subnet:

You need root on each, a current kernel with the vxlan module (any supported Ubuntu, Debian or RHEL release has it), and iproute2. The persistence step uses systemd-networkd. Replace eth0 with your actual underlay interface name.

Step 1: Open UDP 4789

Each VTEP must accept VXLAN from its peers and nobody else. On server-a, with nftables and an existing inet filter table:

sudo nft add rule inet filter input ip saddr { 198.51.100.20, 192.0.2.30 } udp dport 4789 accept

Mirror the rule on the other servers with their peers as sources. Cloud security groups need the same rule. Keep the source list tight, because any host that can reach port 4789 can inject frames into the overlay.

Step 2: Create a point-to-point tunnel

Start with two servers. On server-a:

sudo ip link add vxlan100 type vxlan id 100 dstport 4789 \
    local 203.0.113.10 remote 198.51.100.20 dev eth0
sudo ip addr add 10.200.0.1/24 dev vxlan100
sudo ip link set vxlan100 up

On server-b, swap local and remote and use 10.200.0.2/24.

What the key options do:

• id 100 is the VNI. It must match on both ends.
• dstport 4789 sets the IANA port. Always set it explicitly. The Linux kernel's built-in default is 8472, which predates the IANA assignment, and a mismatch means each side listens on a port the other never sends to.
• local and remote are the underlay addresses of the two VTEPs. A single remote writes an all-zeros FDB entry pointing at the peer, which is all a point-to-point link needs.
• dev eth0 binds the VTEP to the underlay interface, so the kernel derives the MTU from it.

Check the result:

ip -d link show vxlan100

The output should include mtu 1450 and a line like vxlan id 100 remote 198.51.100.20 local 203.0.113.10 dev eth0 ... dstport 4789. If the MTU reads 1500, set it with sudo ip link set vxlan100 mtu 1450.

Step 3: Verify the tunnel

From server-a, ping the overlay address of server-b:

ping -c 3 10.200.0.2

While it runs, capture the underlay on server-b:

sudo tcpdump -ni eth0 udp port 4789

tcpdump decodes VXLAN on its standard port, so each packet shows both layers:

IP 203.0.113.10.51732 > 198.51.100.20.4789: VXLAN, flags [I] (0x08), vni 100
IP 10.200.0.1 > 10.200.0.2: ICMP echo request, id 4, seq 1, length 64

The inner ICMP packet is readable on the wire. That's the expected result, and it's the reason the next sections matter.

Step 4: Add more peers with static FDB entries

A single remote only covers one peer. For three or more, create the interface without remote and write the flood list yourself. On server-a:

sudo ip link del vxlan100
sudo ip link add vxlan100 type vxlan id 100 dstport 4789 local 203.0.113.10 dev eth0
sudo ip addr add 10.200.0.1/24 dev vxlan100
sudo ip link set vxlan100 up

sudo bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 198.51.100.20
sudo bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 192.0.2.30

Each all-zeros entry adds a VTEP to the flood list. When server-a sends an ARP request, the kernel copies it to every listed peer, a pattern called head-end replication. Replies come back as unicast, and with learning on (the default), the kernel records which VTEP each remote MAC lives behind.

Use append, not add. All the flood entries share the same MAC, and add would refuse the second one. Repeat on server-b and server-c, each listing the other two. Inspect the table with:

bridge fdb show dev vxlan100

For a fully static setup, recreate each VXLAN interface with a fixed MAC and learning off. address has to come before type vxlan:

sudo ip link del vxlan100
sudo ip link add vxlan100 address 02:00:00:00:00:01 type vxlan id 100 dstport 4789 \
    local 203.0.113.10 dev eth0 nolearning
sudo ip addr add 10.200.0.1/24 dev vxlan100
sudo ip link set vxlan100 up

Add the flood entries back, plus one unicast entry per peer MAC, such as sudo bridge fdb append 02:00:00:00:00:02 dev vxlan100 dst 198.51.100.20. That trades a little config for predictable forwarding that never depends on learned state.

Step 5: Persist the config with systemd-networkd

Commands from ip are gone after a reboot. This step uses systemd-networkd, which is already running on Ubuntu Server. Elsewhere, start it with sudo systemctl enable --now systemd-networkd. On RHEL it isn't installed by default and comes from EPEL.

On server-a, create /etc/systemd/network/vxlan100.netdev:

[NetDev]
Name=vxlan100
Kind=vxlan

[VXLAN]
VNI=100
Local=203.0.113.10
DestinationPort=4789

Then /etc/systemd/network/vxlan100.network:

[Match]
Name=vxlan100

[Network]
Address=10.200.0.1/24

[BridgeFDB]
MACAddress=00:00:00:00:00:00
Destination=198.51.100.20

[BridgeFDB]
MACAddress=00:00:00:00:00:00
Destination=192.0.2.30

Finally, attach the VXLAN device to the underlay. In the existing .network file that matches eth0, add this line under [Network]:

VXLAN=vxlan100

Delete the manual interface and reload:

sudo ip link del vxlan100
sudo networkctl reload

If eth0 is configured by netplan, NetworkManager or ifupdown rather than a .network file you control, set Independent=yes in the [VXLAN] section so the device doesn't need a parent, and set MTUBytes=1450 under a [Link] section in vxlan100.network.

Repeat on server-b and server-c with their own Local= and Address= values, and the other two peers as Destination= entries.

Encrypting VXLAN traffic

VXLAN has no encryption and no authentication. Anyone on the path can read the inner frames, and anyone who can reach UDP 4789 can inject them. Over the public internet, put it inside an encrypted layer:

• IPsec transport mode. Encrypt all traffic between the VTEPs, and VXLAN is covered with no extra config. See Configuring IPsec.
• WireGuard underneath. Build a WireGuard mesh first, then use the WireGuard addresses as the VXLAN local and remote.

Both add their own overhead, so recompute the VXLAN MTU after stacking them. With WireGuard's default 1420-byte MTU underneath, the VXLAN interface drops to 1370.

Troubleshooting

Start with the counters. ip -s link show vxlan100 shows TX and RX errors, journalctl -u systemd-networkd shows config problems, and tcpdump on the underlay shows what actually leaves the host.

Running VXLAN past a few servers

Static FDB entries scale linearly per host and quadratically across the fleet. Every server lists every other, so adding one server means touching all of them. Head-end replication grows the same way, because each broadcast is copied once per peer.

The usual next step is a real control plane. BGP EVPN, typically run with FRRouting, distributes MAC and VTEP reachability so nobody edits FDB tables by hand. That's a routing protocol to deploy, monitor and debug. The encryption layer, per-path MTU and firewall rules stay yours too.

Each of those is solvable. Together they're a network team's standing workload, and that's the part Cycle's control plane handles for you.

An overlay is easy to build and expensive to keep correct.