The Proxmox VE SDN stack has been available as an experimental feature since 2019 and has been continuously improved and tested by many developers and users. With its integration into the web interface in Proxmox VE 6.2, a significant milestone towards broader integration was achieved. During the Proxmox VE 7 release cycle, numerous improvements and features were added. Based on user feedback, it became apparent that the fundamental design choices and their implementation were quite sound and stable. Consequently, labeling it as ‘experimental’ did not do justice to the state of the SDN stack. For Proxmox VE 8, a decision was made to lay the groundwork for full integration of the SDN feature by elevating the management of networks and interfaces to a core component in the Proxmox VE access control stack. In Proxmox VE 8.1, two major milestones were achieved: firstly, DHCP integration was added to the IP address management (IPAM) feature, and secondly, the SDN integration is now installed by default.

The current support status for the various layers of our SDN installation is as follows:

Since Proxmox VE 8.1 the core Software-Defined Network (SDN) packages are installed by default.

If you upgrade from an older version, you need to install the libpve-network-perl package on every node:

After installation, you need to ensure that the following line is present at the end of the /etc/network/interfaces configuration file on all nodes, so that the SDN configuration gets included and activated.

The DHCP integration into the built-in PVE IP Address Management stack currently uses dnsmasq for giving out DHCP leases. This is currently opt-in.

To use that feature you need to install the dnsmasq package on every node:

The Proxmox VE SDN stack uses the FRRouting project for advanced setups. This is currently opt-in.

To use the SDN routing integration you need to install the frr-pythontools package on all nodes:

Then enable the frr service on all nodes:

The following options are available for all zone types:

This is the simplest plugin. It will create an isolated VNet bridge. This bridge is not linked to a physical interface, and VM traffic is only local on each the node. It can be used in NAT or routed setups.

The VLAN plugin uses an existing local Linux or OVS bridge to connect to the node’s physical interface. It uses VLAN tagging defined in the VNet to isolate the network segments. This allows connectivity of VMs between different nodes.

VLAN zone configuration options:

QinQ also known as VLAN stacking, that uses multiple layers of VLAN tags for isolation. The QinQ zone defines the outer VLAN tag (the Service VLAN) whereas the inner VLAN tag is defined by the VNet.

QinQ zone configuration options:

The VXLAN plugin establishes a tunnel (overlay) on top of an existing network (underlay). This encapsulates layer 2 Ethernet frames within layer 4 UDP datagrams using the default destination port 4789.

You have to configure the underlay network yourself to enable UDP connectivity between all peers.

You can, for example, create a VXLAN overlay network on top of public internet, appearing to the VMs as if they share the same local Layer 2 network.

VXLAN zone configuration options:

The EVPN zone creates a routable Layer 3 network, capable of spanning across multiple clusters. This is achieved by establishing a VPN and utilizing BGP as the routing protocol.

The VNet of EVPN can have an anycast IP address and/or MAC address. The bridge IP is the same on each node, meaning a virtual guest can use this address as gateway.

Routing can work across VNets from different zones through a VRF (Virtual Routing and Forwarding) interface.

EVPN zone configuration options:

The EVPN, zone requires an external controller to manage the control plane. The EVPN controller plugin configures the Free Range Routing (frr) router.

To enable the EVPN controller, you need to enable FRR on every node, see install FRRouting.

EVPN controller configuration options:

The BGP controller is not used directly by a zone. You can use it to configure FRR to manage BGP peers.

For BGP-EVPN, it can be used to define a different ASN by node, so doing EBGP. It can also be used to export EVPN routes to an external BGP peer.

BGP controller configuration options:

The ISIS controller is not used directly by a zone. You can use it to configure FRR to export EVPN routes to an ISIS domain.

ISIS controller configuration options:

The FRR implementations of OpenFabric and OSPF are used, so first ensure that the frr and frr-pythontools packages are installed:

To view the configuration of an SDN fabric users need SDN.Audit or SDN.Allocate permissions. To create or modify a fabric configuration, users need SDN.Allocate permissions. To view the configuration of a node, users need the Sys.Audit or Sys.Modify permissions. When adding or updating nodes within a fabric, additional Sys.Modify permission for the specific node is required, since this operation involves writing to the node’s /etc/network/interfaces file.

To create a Fabric, head over to Datacenter→SDN→Fabrics and click "Add Fabric". After selecting the preferred protocol, the fabric is created. With the "+" button you can select the nodes which you want to add to the fabric, you also have to select the interfaces used to communicate with the other nodes.

OpenFabric is a routing protocol specifically designed for data center fabrics. It’s based on IS-IS and optimized for the spine-leaf topology common in data centers.

Configuration options:

OSPF (Open Shortest Path First) is a widely-used link-state routing protocol that efficiently calculates the shortest path for routing traffic through IP networks.

Configuration options:

WireGuard can be used for establishing a VPN between Proxmox VE nodes and / or external nodes. It does not provide dynamic routing by itself, but can be used in conjunction with dynamic routing protocols operating on layer 3 and above (OSPF, BGP) to provide a dynamically routed, encrypted transport, for example to carry EVPN or VXLAN traffic.

Configuration options:

A prefix list is an ordered, named list of entries that each match a single destination CIDR (or a range of prefix lengths within that CIDR) and either permit or deny it. FRR evaluates entries in order of their sequence number and stops at the first match; if no entry matches, the prefix list denies implicitly, same as plain FRR.

Prefix lists are managed under Datacenter -> SDN -> Prefix Lists. Each list has a name and contains one or more entries with the following properties:

Without le or ge an entry matches the prefix exactly. With both set, it matches any subnet within the given length range. For example 10.0.0.0/8 ge 16 le 24 matches every /16 to /24 subnet inside 10.0.0.0/8. This is the standard FRR/Cisco semantics. Additional validation enforced by Proxmox VE: ge must not exceed le, neither may be smaller than the prefix’s own length, and both must stay within the address family bound (/32 for IPv4, /128 for IPv6).

A route map is an ordered list of entries that match incoming or outgoing routes against criteria, optionally modify attributes such as the metric or local preference, and either permit (forward the route, possibly modified) or deny (drop the route). FRR evaluates entries in order; the first match controls the action. Every route map implicitly has an otherwise empty deny as its last entry, denying all routes that did not match an entry explicitly.

Route maps are managed under Datacenter -> SDN -> Route Maps. Each entry has the following properties:

The BGP and EVPN controllers each take an incoming and an outgoing route map. Incoming maps run against routes received from a peer before installing them; outgoing maps run against routes before announcing them. Both fields are optional and selected from the route maps configured in the SDN.

The OSPF and OpenFabric fabrics take a Route Filter option that references a prefix list. When set, only routes whose destinations pass the prefix list are installed in the kernel routing table. Without an explicit filter, the fabric generates a default filter from the fabric’s own IPv4 Prefix and IPv6 Prefix CIDRs (an FRR access-list permitting those networks); configure a route filter when you need a different policy, for example to allow a default route learned from a border node.

Allow only the IPv4 default route, for example as a route filter on a node that should learn just an upstream default:

Allow every subnet between /16 and /24 inside 10.0.0.0/8, and deny anything else within the 10.0.0.0/8 block:

Outgoing BGP policy that lowers local-preference to 50 on routes matched by the internal-subnets prefix list, and otherwise leaves routes untouched. Each route map entry is its own section, with the order encoded as a suffix on the section ID:

The trailing permit entry with no match is the FRR pattern for "allow everything else"; without it the implicit deny at the end of the route map would drop all unmatched routes.

BGP (Border Gateway Protocol) can be used as an eBGP unnumbered fabric. Each node has its own Autonomous System Number (ASN) and peers with its neighbors over physical interfaces without requiring IP addresses on the fabric links.

Configuration options:

The default built-in IPAM for your Proxmox VE cluster.

You can inspect the current status of the PVE IPAM Plugin via the IPAM panel in the SDN section of the datacenter configuration. This UI can be used to create, update and delete IP mappings. This is particularly convenient in conjunction with the DHCP feature.

If you are using DHCP, you can use the IPAM panel to create or edit leases for specific VMs, which enables you to change the IPs allocated via DHCP. When editing an IP of a VM that is using DHCP you must make sure to force the guest to acquire a new DHCP leases. This can usually be done by reloading the network stack of the guest or rebooting it.

NetBox is an open-source IP Address Management (IPAM) and datacenter infrastructure management (DCIM) tool.

To integrate NetBox with Proxmox VE SDN, create an API token in NetBox as described here: https://docs.netbox.dev/en/stable/integrations/rest-api/#tokens

The NetBox configuration properties are:

In phpIPAM you need to create an "application" and add an API token with admin privileges to the application.

The phpIPAM configuration properties are:

https://doc.powerdns.com/authoritative/http-api/index.html

You need to enable the web server and the API in your PowerDNS config:

The PowerDNS configuration options are:

You can enable automatic DHCP for a zone in the Web UI via the Zones panel and enabling DHCP in the advanced options of a zone.

After automatic DHCP has been enabled for a Zone, DHCP Ranges need to be configured for the subnets in a Zone. In order to that, go to the Vnets panel and select the Subnet for which you want to configure DHCP ranges. In the edit dialogue you can configure DHCP ranges in the respective Tab. Alternatively you can set DHCP ranges for a Subnet via the following CLI command:

You also need to have a gateway configured for the subnet - otherwise automatic DHCP will not work.

The DHCP plugin will then allocate IPs in the IPAM only in the configured ranges.

Do not forget to follow the installation steps for the dnsmasq DHCP plugin as well.

The firewall automatically generates the following IPSets in the SDN scope for every VNet:

When making changes to your configuration, the IPSets update automatically, so you do not have to update your firewall rules when changing the configuration of your Subnets.

If you are using the built-in PVE IPAM, then the firewall automatically generates an IPset for every guest that has entries in the IPAM. The respective IPset for a guest with ID 100 would be guest-ipam-100. It contains all IP addresses from all IPAM entries. So if guest 100 is member of multiple VNets, then the IPset would contain the IPs from all VNets.

When entries get added / updated / deleted, then the respective IPSets will be updated accordingly.

Simple zone networks create an isolated network for guests on a single host to connect to each other.

The network interface configuration in two VMs may look like this which allows them to communicate via the 10.0.1.0/24 network.

If you want to allow outgoing connections for guests in the simple network zone the simple zone offers a Source NAT (SNAT) option.

Starting from the configuration above, Add a Subnet to the VNet vnet1, set a gateway IP and enable the SNAT option.

In the guests configure the static IP address inside the subnet’s IP range.

The node itself will join this network with the Gateway IP 172.16.0.1 and function as the NAT gateway for guests within the subnet range.

When VMs on different nodes need to communicate through an isolated network, the VLAN zone allows network level isolation using VLAN tags.

Create a VLAN zone named myvlanzone:

Create a VNet named myvnet1 with VLAN tag 10 and the previously created myvlanzone.

Apply the configuration through the main SDN panel, to create VNets locally on each node.

Create a Debian-based virtual machine (vm1) on node1, with a vNIC on myvnet1.

Use the following network configuration for this VM:

Create a second virtual machine (vm2) on node2, with a vNIC on the same VNet myvnet1 as vm1.

Use the following network configuration for this VM:

Following this, you should be able to ping between both VMs using that network.

This example configures two QinQ zones and adds two VMs to each zone to demonstrate the additional layer of VLAN tags which allows the configuration of more isolated VLANs.

A typical use case for this configuration is a hosting provider that provides an isolated network to customers for VM communication but isolates the VMs from other customers.

Create a QinQ zone named qinqzone1 with service VLAN 20

Create another QinQ zone named qinqzone2 with service VLAN 30

Create a VNet named myvnet1 with VLAN-ID 100 on the previously created qinqzone1 zone.

Create a myvnet2 with VLAN-ID 100 on the qinqzone2 zone.

Apply the configuration on the main SDN web interface panel to create VNets locally on each node.

Create four Debian-bases virtual machines (vm1, vm2, vm3, vm4) and add network interfaces to vm1 and vm2 with bridge qinqvnet1 and vm3 and vm4 with bridge qinqvnet2.

Inside the VM, configure the IP addresses of the interfaces, for example via /etc/network/interfaces:

Configure all four VMs to have IP addresses from the 10.0.3.101 to 10.0.3.104 range.

Now you should be able to ping between the VMs vm1 and vm2, as well as between vm3 and vm4. However, neither of VMs vm1 or vm2 can ping VMs vm3 or vm4, as they are on a different zone with a different service-VLAN.

The example assumes a cluster with three nodes, with the node IP addresses 192.168.0.1, 192.168.0.2 and 192.168.0.3.

Create a VXLAN zone named myvxlanzone and add all IPs from the nodes to the peer address list. Use the default MTU of 1450 or configure accordingly.

Create a VNet named vxvnet1 using the VXLAN zone myvxlanzone created previously.

Apply the configuration on the main SDN web interface panel to create VNets locally on each nodes.

Create a Debian-based virtual machine (vm1) on node1, with a vNIC on vxvnet1.

Use the following network configuration for this VM (note the lower MTU).

Create a second virtual machine (vm2) on node3, with a vNIC on the same VNet vxvnet1 as vm1.

Use the following network configuration for this VM:

Then, you should be able to ping between between vm1 and vm2.

The example assumes a cluster with three nodes (node1, node2, node3) with IP addresses 192.168.0.1, 192.168.0.2 and 192.168.0.3.

Create an EVPN controller, using a private ASN number and the above node addresses as peers.

Create an EVPN zone named myevpnzone, assign the previously created EVPN-controller and define node1 and node2 as exit nodes.

Create the first VNet named myvnet1 using the EVPN zone myevpnzone.

Create a subnet on myvnet1:

Create the second VNet named myvnet2 using the same EVPN zone myevpnzone.

Create a different subnet on myvnet2`:

Apply the configuration from the main SDN web interface panel to create VNets locally on each node and generate the FRR configuration.

Create a Debian-based virtual machine (vm1) on node1, with a vNIC on myvnet1.

Use the following network configuration for vm1:

Create a second virtual machine (vm2) on node2, with a vNIC on the other VNet myvnet2.

Use the following network configuration for vm2:

Now you should be able to ping vm2 from vm1, and vm1 from vm2.

If you ping an external IP from vm2 on the non-gateway node3, the packet will go to the configured myvnet2 gateway, then will be routed to the exit nodes (node1 or node2) and from there it will leave those nodes over the default gateway configured on node1 or node2.

If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24 and 10.0.2.0/24 in this example), will be announced dynamically.

If you have multiple gateway nodes, you should disable the rp_filter (Strict Reverse Path Filter) option, because packets can arrive at one node but go out from another node.

Create a drop-in sysctl file /etc/sysctl.d/99-rp-filter.conf:

To add IPSEC encryption on top of a VXLAN, this example shows how to use strongswan.

You`ll need to reduce the MTU by additional 60 bytes for IPv4 or 80 bytes for IPv6 to handle encryption.

So with default real 1500 MTU, you need to use a MTU of 1370 (1370 + 80 (IPSEC) + 50 (VXLAN) == 1500).

Install strongswan on the host.

Add configuration to /etc/ipsec.conf. We only need to encrypt traffic from the VXLAN UDP port 4789.

Generate a pre-shared key with:

and add the key to /etc/ipsec.secrets, so that the file contents looks like:

Copy the PSK and the configuration to all nodes participating in the VXLAN network.