三层扩展
三层扩展使用 EVPN 作为控制平面，类似于使用 VXLAN 隧道的三层 VPN。叶交换机在 POD 内部和跨 POD 建立全网状 VXLAN 隧道，POD 之间的路由交换通过 EVPN type-5 路由进行。在 POD 内部，有用于 EVPN 多宿主的 type-1 和 type-4 路由，用于 MAC 地址、IP 地址和 MAC 路由的 type-2 路由，以及用于 BUM HER 路由的 type-3 路由。
配置
| DCI 1 | DCI 2 | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
| ||||||||||||
|
|
以下配置示例使用下游 VNI 和对称路由将 DC1 中的 VRF RED 与 DC2 中的 VRF RED 连接起来。`route-target import` 语句在三层连接两个 RED VRF（用于前缀交换）。此配置提供 VRF RED 内的 server01 和 server03 之间，以及 VRF GREEN 内的 server02 和 server04 之间的 IP 连接，但 RED 和 GREEN VRF 之间无法通信。所有服务器都位于不同的 IP 子网中；它们之间没有二层邻接关系。服务器通过其默认网关（即 ARP 缓存中的本地 VRR MAC 地址）与其在另一个 DC 中的对等服务器通信。
该示例展示了一个三层互连配置，其中边界叶节点过滤 EVPN 前缀（type-5 除外）以在 DCI 链路上传输。此配置确保 DCI 仅交换 type-5 前缀，并且远程 DC 不接收和处理不需要的前缀类型。ESI 和 MAC 地址对于每个本地 POD 是可见的，但跨 POD 不可见。
ubuntu@server01:~$ cat /etc/netplan/config.yaml
################################################################
# IMPORTANT: When using NVIDIA Air services, #
# your Internet-facing interface must include the following: #
# dhcp6: false #
# accept-ra: true #
################################################################
network:
version: 2
renderer: networkd
ethernets:
eth0:
dhcp4: true
eth1:
dhcp4: false
eth2:
dhcp4: false
bonds:
bond0:
interfaces: [eth1, eth2]
addresses: [192.168.1.10/24]
gateway4: 192.168.1.1
routes:
- to: 192.168.0.0/16
via: 192.168.1.1
parameters:
mode: 802.3ad
version: 2
ubuntu@server01:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 44:38:39:22:aa:01 brd ff:ff:ff:ff:ff:ff
inet 192.168.200.20/24 brd 192.168.200.255 scope global dynamic eth0
valid_lft 163305sec preferred_lft 163305sec
inet6 fe80::4638:39ff:fe22:aa01/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond0 state UP group default qlen 1000
link/ether 42:20:47:91:95:a7 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond0 state UP group default qlen 1000
link/ether 42:20:47:91:95:a7 brd ff:ff:ff:ff:ff:ff
5: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 42:20:47:91:95:a7 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.10/24 brd 192.168.1.255 scope global bond0
valid_lft forever preferred_lft forever
inet6 fe80::4020:47ff:fe91:95a7/64 scope link
valid_lft forever preferred_lft forever
ubuntu@server03:~$ cat /etc/netplan/config.yaml
################################################################
# IMPORTANT: When using NVIDIA Air services, #
# your Internet-facing interface must include the following: #
# dhcp6: false #
# accept-ra: true #
################################################################
network:
version: 2
renderer: networkd
ethernets:
eth0:
dhcp4: true
eth1:
dhcp4: false
eth2:
dhcp4: false
bonds:
bond0:
interfaces: [eth1, eth2]
addresses: [192.168.10.110/24]
gateway4: 192.168.10.100
routes:
- to: 192.168.0.0/16
via: 192.168.10.100
parameters:
mode: 802.3ad
version: 2
ubuntu@server03:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 44:38:39:22:aa:03 brd ff:ff:ff:ff:ff:ff
inet 192.168.200.22/24 brd 192.168.200.255 scope global dynamic eth0
valid_lft 163264sec preferred_lft 163264sec
inet6 fe80::4638:39ff:fe22:aa03/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond0 state UP group default qlen 1000
link/ether b6:4b:0f:ea:f2:02 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond0 state UP group default qlen 1000
link/ether b6:4b:0f:ea:f2:02 brd ff:ff:ff:ff:ff:ff
5: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether b6:4b:0f:ea:f2:02 brd ff:ff:ff:ff:ff:ff
inet 192.168.10.110/24 brd 192.168.10.255 scope global bond0
valid_lft forever preferred_lft forever
inet6 fe80::b44b:fff:feea:f202/64 scope link
valid_lft forever preferred_lft forever
cumulus@leaf01:mgmt:~$ nv set bridge domain br_default vlan 10 vni 10
cumulus@leaf01:mgmt:~$ nv set bridge domain br_default vlan 20 vni 20
cumulus@leaf01:mgmt:~$ nv set evpn enable on
cumulus@leaf01:mgmt:~$ nv set evpn multihoming enable on
cumulus@leaf01:mgmt:~$ nv set interface bond1 bond member swp3
cumulus@leaf01:mgmt:~$ nv set interface bond1 bridge domain br_default access 10
cumulus@leaf01:mgmt:~$ nv set interface bond1 evpn multihoming segment local-id 1
cumulus@leaf01:mgmt:~$ nv set interface bond1-2 bond lacp-bypass on
cumulus@leaf01:mgmt:~$ nv set interface bond1-2 evpn multihoming segment df-preference 50000
cumulus@leaf01:mgmt:~$ nv set interface bond1-2 evpn multihoming segment enable on
cumulus@leaf01:mgmt:~$ nv set interface bond1-2 evpn multihoming segment mac-address 00:00:00:00:00:AA
cumulus@leaf01:mgmt:~$ nv set interface bond1-2 type bond
cumulus@leaf01:mgmt:~$ nv set interface bond2 bond member swp4
cumulus@leaf01:mgmt:~$ nv set interface bond2 bridge domain br_default access 20
cumulus@leaf01:mgmt:~$ nv set interface bond2 evpn multihoming segment local-id 2
cumulus@leaf01:mgmt:~$ nv set interface eth0 ip vrf mgmt
cumulus@leaf01:mgmt:~$ nv set interface eth0 type eth
cumulus@leaf01:mgmt:~$ nv set interface lo ip address 10.10.10.1/32
cumulus@leaf01:mgmt:~$ nv set interface lo type loopback
cumulus@leaf01:mgmt:~$ nv set interface swp1-2 evpn multihoming uplink on
cumulus@leaf01:mgmt:~$ nv set interface swp1-2 type swp
cumulus@leaf01:mgmt:~$ nv set interface vlan10 ip address 192.168.1.2/24
cumulus@leaf01:mgmt:~$ nv set interface vlan10 ip vrf RED
cumulus@leaf01:mgmt:~$ nv set interface vlan10 ip vrr address 192.168.1.1/24
cumulus@leaf01:mgmt:~$ nv set interface vlan10 vlan 10
cumulus@leaf01:mgmt:~$ nv set interface vlan10,20 ip vrr enable on
cumulus@leaf01:mgmt:~$ nv set interface vlan10,20 ip vrr state up
cumulus@leaf01:mgmt:~$ nv set interface vlan10,20 type svi
cumulus@leaf01:mgmt:~$ nv set interface vlan20 ip address 192.168.2.2/24
cumulus@leaf01:mgmt:~$ nv set interface vlan20 ip vrf GREEN
cumulus@leaf01:mgmt:~$ nv set interface vlan20 ip vrr address 192.168.2.1/24
cumulus@leaf01:mgmt:~$ nv set interface vlan20 vlan 20
cumulus@leaf01:mgmt:~$ nv set nve vxlan arp-nd-suppress on
cumulus@leaf01:mgmt:~$ nv set nve vxlan enable on
cumulus@leaf01:mgmt:~$ nv set nve vxlan source address 10.10.10.1
cumulus@leaf01:mgmt:~$ nv set router bgp autonomous-system 65101
cumulus@leaf01:mgmt:~$ nv set router bgp enable on
cumulus@leaf01:mgmt:~$ nv set router bgp router-id 10.10.10.1
cumulus@leaf01:mgmt:~$ nv set router vrr enable on
cumulus@leaf01:mgmt:~$ nv set service lldp
cumulus@leaf01:mgmt:~$ nv set system config auto-save enable on
cumulus@leaf01:mgmt:~$ nv set system global anycast-id 10
cumulus@leaf01:mgmt:~$ nv set system global fabric-id 10
cumulus@leaf01:mgmt:~$ nv set system hostname leaf01
cumulus@leaf01:mgmt:~$ nv set vrf GREEN evpn enable on
cumulus@leaf01:mgmt:~$ nv set vrf GREEN evpn vni 4002
cumulus@leaf01:mgmt:~$ nv set vrf GREEN router bgp address-family ipv4-unicast enable on
cumulus@leaf01:mgmt:~$ nv set vrf GREEN router bgp address-family ipv4-unicast redistribute connected enable on
cumulus@leaf01:mgmt:~$ nv set vrf GREEN router bgp address-family ipv4-unicast route-export to-evpn enable on
cumulus@leaf01:mgmt:~$ nv set vrf GREEN router bgp autonomous-system 65101
cumulus@leaf01:mgmt:~$ nv set vrf GREEN router bgp enable on
cumulus@leaf01:mgmt:~$ nv set vrf GREEN router bgp route-import from-evpn route-target ANY:4002
cumulus@leaf01:mgmt:~$ nv set vrf GREEN router bgp route-import from-evpn route-target ANY:5002
cumulus@leaf01:mgmt:~$ nv set vrf GREEN router bgp router-id 10.10.10.1
cumulus@leaf01:mgmt:~$ nv set vrf RED evpn enable on
cumulus@leaf01:mgmt:~$ nv set vrf RED evpn vni 4001
cumulus@leaf01:mgmt:~$ nv set vrf RED router bgp address-family ipv4-unicast enable on
cumulus@leaf01:mgmt:~$ nv set vrf RED router bgp address-family ipv4-unicast redistribute connected enable on
cumulus@leaf01:mgmt:~$ nv set vrf RED router bgp address-family ipv4-unicast route-export to-evpn enable on
cumulus@leaf01:mgmt:~$ nv set vrf RED router bgp autonomous-system 65101
cumulus@leaf01:mgmt:~$ nv set vrf RED router bgp enable on
cumulus@leaf01:mgmt:~$ nv set vrf RED router bgp route-import from-evpn route-target ANY:4001
cumulus@leaf01:mgmt:~$ nv set vrf RED router bgp route-import from-evpn route-target ANY:5001
cumulus@leaf01:mgmt:~$ nv set vrf RED router bgp router-id 10.10.10.1
cumulus@leaf01:mgmt:~$ nv set vrf default router bgp address-family ipv4-unicast enable on
cumulus@leaf01:mgmt:~$ nv set vrf default router bgp address-family ipv4-unicast network 10.10.10.1/32
cumulus@leaf01:mgmt:~$ nv set vrf default router bgp address-family ipv4-unicast redistribute connected enable on
cumulus@leaf01:mgmt:~$ nv set vrf default router bgp address-family l2vpn-evpn enable on
cumulus@leaf01:mgmt:~$ nv set vrf default router bgp enable on
cumulus@leaf01:mgmt:~$ nv set vrf default router bgp neighbor swp1 peer-group underlay
cumulus@leaf01:mgmt:~$ nv set vrf default router bgp neighbor swp1 type unnumbered
cumulus@leaf01:mgmt:~$ nv set vrf default router bgp neighbor swp2 peer-group underlay
cumulus@leaf01:mgmt:~$ nv set vrf default router bgp neighbor swp2 type unnumbered
cumulus@leaf01:mgmt:~$ nv set vrf default router bgp peer-group underlay address-family l2vpn-evpn enable on
cumulus@leaf01:mgmt:~$ nv set vrf default router bgp peer-group underlay remote-as external
cumulus@spine01:mgmt:~$ nv set interface eth0 ip vrf mgmt
cumulus@spine01:mgmt:~$ nv set interface eth0 type eth
cumulus@spine01:mgmt:~$ nv set interface lo ip address 10.10.10.101/32
cumulus@spine01:mgmt:~$ nv set interface lo type loopback
cumulus@spine01:mgmt:~$ nv set interface swp1-4 type swp
cumulus@spine01:mgmt:~$ nv set router bgp autonomous-system 65199
cumulus@spine01:mgmt:~$ nv set router bgp enable on
cumulus@spine01:mgmt:~$ nv set router bgp router-id 10.10.10.101
cumulus@spine01:mgmt:~$ nv set system config auto-save enable on
cumulus@spine01:mgmt:~$ nv set system hostname spine01
cumulus@spine01:mgmt:~$ nv set vrf default router bgp address-family ipv4-unicast enable on
cumulus@spine01:mgmt:~$ nv set vrf default router bgp address-family ipv4-unicast network 10.10.10.101/32
cumulus@spine01:mgmt:~$ nv set vrf default router bgp address-family l2vpn-evpn enable on
cumulus@spine01:mgmt:~$ nv set vrf default router bgp enable on
cumulus@spine01:mgmt:~$ nv set vrf default router bgp neighbor swp1-4 address-family l2vpn-evpn enable on
cumulus@spine01:mgmt:~$ nv set vrf default router bgp neighbor swp1-4 remote-as external
cumulus@spine01:mgmt:~$ nv set vrf default router bgp neighbor swp1-4 type unnumbered
cumulus@borderleaf01:mgmt:~$ nv set evpn enable on
cumulus@borderleaf01:mgmt:~$ nv set interface eth0 ip vrf mgmt
cumulus@borderleaf01:mgmt:~$ nv set interface eth0 type eth
cumulus@borderleaf01:mgmt:~$ nv set interface lo ip address 10.10.10.10/32
cumulus@borderleaf01:mgmt:~$ nv set interface lo type loopback
cumulus@borderleaf01:mgmt:~$ nv set interface swp1-3 type swp
cumulus@borderleaf01:mgmt:~$ nv set nve vxlan enable on
cumulus@borderleaf01:mgmt:~$ nv set router bgp autonomous-system 65110
cumulus@borderleaf01:mgmt:~$ nv set router bgp enable on
cumulus@borderleaf01:mgmt:~$ nv set router bgp router-id 10.10.10.10
cumulus@borderleaf01:mgmt:~$ nv set router policy route-map control_t5 rule 1 action permit
cumulus@borderleaf01:mgmt:~$ nv set router policy route-map control_t5 rule 1 match evpn-route-type ip-prefix
cumulus@borderleaf01:mgmt:~$ nv set router policy route-map control_t5 rule 2 action deny
cumulus@borderleaf01:mgmt:~$ nv set service lldp
cumulus@borderleaf01:mgmt:~$ nv set system config auto-save enable on
cumulus@borderleaf01:mgmt:~$ nv set system global anycast-id 10
cumulus@borderleaf01:mgmt:~$ nv set system global fabric-id 10
cumulus@borderleaf01:mgmt:~$ nv set system hostname borderleaf01
cumulus@borderleaf01:mgmt:~$ nv set vrf GREEN evpn enable on
cumulus@borderleaf01:mgmt:~$ nv set vrf GREEN evpn vni 4002
cumulus@borderleaf01:mgmt:~$ nv set vrf GREEN router bgp address-family ipv4-unicast enable on
cumulus@borderleaf01:mgmt:~$ nv set vrf GREEN router bgp address-family ipv4-unicast redistribute connected enable on
cumulus@borderleaf01:mgmt:~$ nv set vrf GREEN router bgp address-family ipv4-unicast route-export to-evpn enable on
cumulus@borderleaf01:mgmt:~$ nv set vrf GREEN router bgp autonomous-system 65110
cumulus@borderleaf01:mgmt:~$ nv set vrf GREEN router bgp enable on
cumulus@borderleaf01:mgmt:~$ nv set vrf GREEN router bgp peer-group underlay address-family l2vpn-evpn enable on
cumulus@borderleaf01:mgmt:~$ nv set vrf GREEN router bgp route-import from-evpn route-target ANY:4002
cumulus@borderleaf01:mgmt:~$ nv set vrf GREEN router bgp route-import from-evpn route-target ANY:5002
cumulus@borderleaf01:mgmt:~$ nv set vrf GREEN router bgp router-id 10.10.10.10
cumulus@borderleaf01:mgmt:~$ nv set vrf RED evpn enable on
cumulus@borderleaf01:mgmt:~$ nv set vrf RED evpn vni 4001
cumulus@borderleaf01:mgmt:~$ nv set vrf RED router bgp address-family ipv4-unicast enable on
cumulus@borderleaf01:mgmt:~$ nv set vrf RED router bgp address-family ipv4-unicast redistribute connected enable on
cumulus@borderleaf01:mgmt:~$ nv set vrf RED router bgp address-family ipv4-unicast route-export to-evpn enable on
cumulus@borderleaf01:mgmt:~$ nv set vrf RED router bgp autonomous-system 65110
cumulus@borderleaf01:mgmt:~$ nv set vrf RED router bgp enable on
cumulus@borderleaf01:mgmt:~$ nv set vrf RED router bgp peer-group underlay address-family l2vpn-evpn enable on
cumulus@borderleaf01:mgmt:~$ nv set vrf RED router bgp route-import from-evpn route-target ANY:4001
cumulus@borderleaf01:mgmt:~$ nv set vrf RED router bgp route-import from-evpn route-target ANY:5001
cumulus@borderleaf01:mgmt:~$ nv set vrf RED router bgp router-id 10.10.10.10
cumulus@borderleaf01:mgmt:~$ nv set vrf default router bgp address-family ipv4-unicast enable on
cumulus@borderleaf01:mgmt:~$ nv set vrf default router bgp address-family ipv4-unicast network 10.10.10.10/32
cumulus@borderleaf01:mgmt:~$ nv set vrf default router bgp address-family ipv4-unicast redistribute connected enable on
cumulus@borderleaf01:mgmt:~$ nv set vrf default router bgp address-family l2vpn-evpn enable on
cumulus@borderleaf01:mgmt:~$ nv set vrf default router bgp enable on
cumulus@borderleaf01:mgmt:~$ nv set vrf default router bgp neighbor swp1 peer-group underlay
cumulus@borderleaf01:mgmt:~$ nv set vrf default router bgp neighbor swp1 type unnumbered
cumulus@borderleaf01:mgmt:~$ nv set vrf default router bgp neighbor swp2 peer-group underlay
cumulus@borderleaf01:mgmt:~$ nv set vrf default router bgp neighbor swp2 type unnumbered
cumulus@borderleaf01:mgmt:~$ nv set vrf default router bgp neighbor swp3 peer-group dci_group1
cumulus@borderleaf01:mgmt:~$ nv set vrf default router bgp neighbor swp3 type unnumbered
cumulus@borderleaf01:mgmt:~$ nv set vrf default router bgp peer-group dci_group1 address-family l2vpn-evpn enable on
cumulus@borderleaf01:mgmt:~$ nv set vrf default router bgp peer-group dci_group1 address-family l2vpn-evpn policy outbound route-map control_t5
cumulus@borderleaf01:mgmt:~$ nv set vrf default router bgp peer-group dci_group1 remote-as external
cumulus@borderleaf01:mgmt:~$ nv set vrf default router bgp peer-group underlay address-family l2vpn-evpn enable on
cumulus@borderleaf01:mgmt:~$ nv set vrf default router bgp peer-group underlay remote-as external
cumulus@leaf03:mgmt:~$ nv set bridge domain br_default vlan 1010 vni 1010
cumulus@leaf03:mgmt:~$ nv set bridge domain br_default vlan 2020 vni 2020
cumulus@leaf03:mgmt:~$ nv set evpn enable on
cumulus@leaf03:mgmt:~$ nv set evpn multihoming enable on
cumulus@leaf03:mgmt:~$ nv set interface bond1 bond member swp3
cumulus@leaf03:mgmt:~$ nv set interface bond1 bridge domain br_default access 1010
cumulus@leaf03:mgmt:~$ nv set interface bond1 evpn multihoming segment local-id 1
cumulus@leaf03:mgmt:~$ nv set interface bond1-2 bond lacp-bypass on
cumulus@leaf03:mgmt:~$ nv set interface bond1-2 evpn multihoming segment df-preference 50000
cumulus@leaf03:mgmt:~$ nv set interface bond1-2 evpn multihoming segment enable on
cumulus@leaf03:mgmt:~$ nv set interface bond1-2 evpn multihoming segment mac-address 00:00:00:00:00:BB
cumulus@leaf03:mgmt:~$ nv set interface bond1-2 type bond
cumulus@leaf03:mgmt:~$ nv set interface bond2 bond member swp4
cumulus@leaf03:mgmt:~$ nv set interface bond2 bridge domain br_default access 2020
cumulus@leaf03:mgmt:~$ nv set interface bond2 evpn multihoming segment local-id 2
cumulus@leaf03:mgmt:~$ nv set interface eth0 ip vrf mgmt
cumulus@leaf03:mgmt:~$ nv set interface eth0 type eth
cumulus@leaf03:mgmt:~$ nv set interface lo ip address 10.10.20.1/32
cumulus@leaf03:mgmt:~$ nv set interface lo type loopback
cumulus@leaf03:mgmt:~$ nv set interface swp1-2 evpn multihoming uplink on
cumulus@leaf03:mgmt:~$ nv set interface swp1-2 type swp
cumulus@leaf03:mgmt:~$ nv set interface vlan1010 ip address 192.168.10.101/24
cumulus@leaf03:mgmt:~$ nv set interface vlan1010 ip vrf RED
cumulus@leaf03:mgmt:~$ nv set interface vlan1010 ip vrr address 192.168.10.100/24
cumulus@leaf03:mgmt:~$ nv set interface vlan1010 vlan 1010
cumulus@leaf03:mgmt:~$ nv set interface vlan1010,2020 ip vrr enable on
cumulus@leaf03:mgmt:~$ nv set interface vlan1010,2020 ip vrr state up
cumulus@leaf03:mgmt:~$ nv set interface vlan1010,2020 type svi
cumulus@leaf03:mgmt:~$ nv set interface vlan2020 ip address 192.168.20.101/24
cumulus@leaf03:mgmt:~$ nv set interface vlan2020 ip vrf GREEN
cumulus@leaf03:mgmt:~$ nv set interface vlan2020 ip vrr address 192.168.20.100/24
cumulus@leaf03:mgmt:~$ nv set interface vlan2020 vlan 2020
cumulus@leaf03:mgmt:~$ nv set nve vxlan arp-nd-suppress on
cumulus@leaf03:mgmt:~$ nv set nve vxlan enable on
cumulus@leaf03:mgmt:~$ nv set nve vxlan source address 10.10.20.1
cumulus@leaf03:mgmt:~$ nv set router bgp autonomous-system 65201
cumulus@leaf03:mgmt:~$ nv set router bgp enable on
cumulus@leaf03:mgmt:~$ nv set router bgp router-id 10.10.20.1
cumulus@leaf03:mgmt:~$ nv set router vrr enable on
cumulus@leaf03:mgmt:~$ nv set service lldp
cumulus@leaf03:mgmt:~$ nv set system config auto-save enable on
cumulus@leaf03:mgmt:~$ nv set system global anycast-id 20
cumulus@leaf03:mgmt:~$ nv set system global fabric-id 20
cumulus@leaf03:mgmt:~$ nv set system hostname leaf03
cumulus@leaf03:mgmt:~$ nv set vrf GREEN evpn enable on
cumulus@leaf03:mgmt:~$ nv set vrf GREEN evpn vni 5002
cumulus@leaf03:mgmt:~$ nv set vrf GREEN router bgp address-family ipv4-unicast enable on
cumulus@leaf03:mgmt:~$ nv set vrf GREEN router bgp address-family ipv4-unicast redistribute connected enable on
cumulus@leaf03:mgmt:~$ nv set vrf GREEN router bgp address-family ipv4-unicast route-export to-evpn enable on
cumulus@leaf03:mgmt:~$ nv set vrf GREEN router bgp autonomous-system 65201
cumulus@leaf03:mgmt:~$ nv set vrf GREEN router bgp enable on
cumulus@leaf03:mgmt:~$ nv set vrf GREEN router bgp route-import from-evpn route-target ANY:4002
cumulus@leaf03:mgmt:~$ nv set vrf GREEN router bgp route-import from-evpn route-target ANY:5002
cumulus@leaf03:mgmt:~$ nv set vrf GREEN router bgp router-id 10.10.20.1
cumulus@leaf03:mgmt:~$ nv set vrf RED evpn enable on
cumulus@leaf03:mgmt:~$ nv set vrf RED evpn vni 5001
cumulus@leaf03:mgmt:~$ nv set vrf RED router bgp address-family ipv4-unicast enable on
cumulus@leaf03:mgmt:~$ nv set vrf RED router bgp address-family ipv4-unicast redistribute connected enable on
cumulus@leaf03:mgmt:~$ nv set vrf RED router bgp address-family ipv4-unicast route-export to-evpn enable on
cumulus@leaf03:mgmt:~$ nv set vrf RED router bgp autonomous-system 65201
cumulus@leaf03:mgmt:~$ nv set vrf RED router bgp enable on
cumulus@leaf03:mgmt:~$ nv set vrf RED router bgp route-import from-evpn route-target ANY:4001
cumulus@leaf03:mgmt:~$ nv set vrf RED router bgp route-import from-evpn route-target ANY:5001
cumulus@leaf03:mgmt:~$ nv set vrf RED router bgp router-id 10.10.20.1
cumulus@leaf03:mgmt:~$ nv set vrf default router bgp address-family ipv4-unicast enable on
cumulus@leaf03:mgmt:~$ nv set vrf default router bgp address-family ipv4-unicast network 10.10.20.1/32
cumulus@leaf03:mgmt:~$ nv set vrf default router bgp address-family ipv4-unicast redistribute connected enable on
cumulus@leaf03:mgmt:~$ nv set vrf default router bgp address-family l2vpn-evpn enable on
cumulus@leaf03:mgmt:~$ nv set vrf default router bgp enable on
cumulus@leaf03:mgmt:~$ nv set vrf default router bgp neighbor swp1 address-family l2vpn-evpn enable on
cumulus@leaf03:mgmt:~$ nv set vrf default router bgp neighbor swp1 remote-as external
cumulus@leaf03:mgmt:~$ nv set vrf default router bgp neighbor swp1 type unnumbered
cumulus@leaf03:mgmt:~$ nv set vrf default router bgp neighbor swp2 address-family l2vpn-evpn enable on
cumulus@leaf03:mgmt:~$ nv set vrf default router bgp neighbor swp2 remote-as external
cumulus@leaf03:mgmt:~$ nv set vrf default router bgp neighbor swp2 type unnumbered
cumulus@spine03:mgmt:~$ nv set interface eth0 ip vrf mgmt
cumulus@spine03:mgmt:~$ nv set interface eth0 type eth
cumulus@spine03:mgmt:~$ nv set interface lo ip address 10.10.20.103/32
cumulus@spine03:mgmt:~$ nv set interface lo type loopback
cumulus@spine03:mgmt:~$ nv set interface swp1-4 type swp
cumulus@spine03:mgmt:~$ nv set router bgp autonomous-system 65299
cumulus@spine03:mgmt:~$ nv set router bgp enable on
cumulus@spine03:mgmt:~$ nv set router bgp router-id 10.10.20.103
cumulus@spine03:mgmt:~$ nv set system config auto-save enable on
cumulus@spine03:mgmt:~$ nv set system global anycast-id 20
cumulus@spine03:mgmt:~$ nv set system global fabric-id 20
cumulus@spine03:mgmt:~$ nv set system hostname spine03
cumulus@spine03:mgmt:~$ nv set vrf default router bgp address-family ipv4-unicast enable on
cumulus@spine03:mgmt:~$ nv set vrf default router bgp address-family ipv4-unicast network 10.10.20.103/32
cumulus@spine03:mgmt:~$ nv set vrf default router bgp address-family l2vpn-evpn enable on
cumulus@spine03:mgmt:~$ nv set vrf default router bgp enable on
cumulus@spine03:mgmt:~$ nv set vrf default router bgp neighbor swp1-4 address-family l2vpn-evpn enable on
cumulus@spine03:mgmt:~$ nv set vrf default router bgp neighbor swp1-4 remote-as external
cumulus@spine03:mgmt:~$ nv set vrf default router bgp neighbor swp1-4 type unnumbered
cumulus@borderleaf04:mgmt:~$ nv set evpn enable on
cumulus@borderleaf04:mgmt:~$ nv set interface eth0 ip vrf mgmt
cumulus@borderleaf04:mgmt:~$ nv set interface eth0 type eth
cumulus@borderleaf04:mgmt:~$ nv set interface lo ip address 10.10.20.11/32
cumulus@borderleaf04:mgmt:~$ nv set interface lo type loopback
cumulus@borderleaf04:mgmt:~$ nv set interface swp1-3 type swp
cumulus@borderleaf04:mgmt:~$ nv set nve vxlan enable on
cumulus@borderleaf04:mgmt:~$ nv set router bgp autonomous-system 65210
cumulus@borderleaf04:mgmt:~$ nv set router bgp enable on
cumulus@borderleaf04:mgmt:~$ nv set router bgp router-id 10.10.20.11
cumulus@borderleaf04:mgmt:~$ nv set router policy route-map control_t5 rule 1 action permit
cumulus@borderleaf04:mgmt:~$ nv set router policy route-map control_t5 rule 1 match evpn-route-type ip-prefix
cumulus@borderleaf04:mgmt:~$ nv set router policy route-map control_t5 rule 2 action deny
cumulus@borderleaf04:mgmt:~$ nv set service lldp
cumulus@borderleaf04:mgmt:~$ nv set system config auto-save enable on
cumulus@borderleaf04:mgmt:~$ nv set system global anycast-id 20
cumulus@borderleaf04:mgmt:~$ nv set system global fabric-id 20
cumulus@borderleaf04:mgmt:~$ nv set system hostname borderleaf04
cumulus@borderleaf04:mgmt:~$ nv set vrf GREEN evpn enable on
cumulus@borderleaf04:mgmt:~$ nv set vrf GREEN evpn vni 5002
cumulus@borderleaf04:mgmt:~$ nv set vrf GREEN router bgp address-family ipv4-unicast enable on
cumulus@borderleaf04:mgmt:~$ nv set vrf GREEN router bgp address-family ipv4-unicast redistribute connected enable on
cumulus@borderleaf04:mgmt:~$ nv set vrf GREEN router bgp address-family ipv4-unicast route-export to-evpn enable on
cumulus@borderleaf04:mgmt:~$ nv set vrf GREEN router bgp autonomous-system 65210
cumulus@borderleaf04:mgmt:~$ nv set vrf GREEN router bgp enable on
cumulus@borderleaf04:mgmt:~$ nv set vrf GREEN router bgp peer-group underlay address-family l2vpn-evpn enable on
cumulus@borderleaf04:mgmt:~$ nv set vrf GREEN router bgp route-import from-evpn route-target ANY:4002
cumulus@borderleaf04:mgmt:~$ nv set vrf GREEN router bgp route-import from-evpn route-target ANY:5002
cumulus@borderleaf04:mgmt:~$ nv set vrf GREEN router bgp router-id 10.10.20.11
cumulus@borderleaf04:mgmt:~$ nv set vrf RED evpn enable on
cumulus@borderleaf04:mgmt:~$ nv set vrf RED evpn vni 5001
cumulus@borderleaf04:mgmt:~$ nv set vrf RED router bgp address-family ipv4-unicast enable on
cumulus@borderleaf04:mgmt:~$ nv set vrf RED router bgp address-family ipv4-unicast redistribute connected enable on
cumulus@borderleaf04:mgmt:~$ nv set vrf RED router bgp address-family ipv4-unicast route-export to-evpn enable on
cumulus@borderleaf04:mgmt:~$ nv set vrf RED router bgp autonomous-system 65210
cumulus@borderleaf04:mgmt:~$ nv set vrf RED router bgp enable on
cumulus@borderleaf04:mgmt:~$ nv set vrf RED router bgp peer-group underlay address-family l2vpn-evpn enable on
cumulus@borderleaf04:mgmt:~$ nv set vrf RED router bgp route-import from-evpn route-target ANY:4001
cumulus@borderleaf04:mgmt:~$ nv set vrf RED router bgp route-import from-evpn route-target ANY:5001
cumulus@borderleaf04:mgmt:~$ nv set vrf RED router bgp router-id 10.10.20.11
cumulus@borderleaf04:mgmt:~$ nv set vrf default router bgp address-family ipv4-unicast enable on
cumulus@borderleaf04:mgmt:~$ nv set vrf default router bgp address-family ipv4-unicast network 10.10.20.11/32
cumulus@borderleaf04:mgmt:~$ nv set vrf default router bgp address-family ipv4-unicast redistribute connected enable on
cumulus@borderleaf04:mgmt:~$ nv set vrf default router bgp address-family l2vpn-evpn enable on
cumulus@borderleaf04:mgmt:~$ nv set vrf default router bgp enable on
cumulus@borderleaf04:mgmt:~$ nv set vrf default router bgp neighbor swp1 peer-group underlay
cumulus@borderleaf04:mgmt:~$ nv set vrf default router bgp neighbor swp1 type unnumbered
cumulus@borderleaf04:mgmt:~$ nv set vrf default router bgp neighbor swp2 peer-group underlay
cumulus@borderleaf04:mgmt:~$ nv set vrf default router bgp neighbor swp2 type unnumbered
cumulus@borderleaf04:mgmt:~$ nv set vrf default router bgp neighbor swp3 peer-group dci_group1
cumulus@borderleaf04:mgmt:~$ nv set vrf default router bgp neighbor swp3 type unnumbered
cumulus@borderleaf04:mgmt:~$ nv set vrf default router bgp peer-group dci_group1 address-family l2vpn-evpn enable on
cumulus@borderleaf04:mgmt:~$ nv set vrf default router bgp peer-group dci_group1 address-family l2vpn-evpn policy outbound route-map control_t5
cumulus@borderleaf04:mgmt:~$ nv set vrf default router bgp peer-group dci_group1 remote-as external
cumulus@borderleaf04:mgmt:~$ nv set vrf default router bgp peer-group underlay address-family l2vpn-evpn enable on
cumulus@borderleaf04:mgmt:~$ nv set vrf default router bgp peer-group underlay remote-as external
诊断命令
以下示例显示了故障排除命令。
cumulus@leaf01:mgmt:~$ net show bgp sum
show bgp ipv4 unicast summary
=============================
BGP router identifier 10.10.10.1, local AS number 65101 vrf-id 0
BGP table version 216
RIB entries 23, using 4600 bytes of memory
Peers 2, using 46 KiB of memory
Peer groups 1, using 64 bytes of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
spine01(swp1) 4 65199 2924 2922 0 0 0 02:18:51 10 12
spine02(swp2) 4 65199 2924 2922 0 0 0 02:18:51 10 12
Total number of neighbors 2
show bgp ipv6 unicast summary
=============================
% No BGP neighbors found
show bgp l2vpn evpn summary
===========================
BGP router identifier 10.10.10.1, local AS number 65101 vrf-id 0
BGP table version 0
RIB entries 75, using 15 KiB of memory
Peers 2, using 46 KiB of memory
Peer groups 1, using 64 bytes of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
spine01(swp1) 4 65199 2924 2922 0 0 0 02:18:52 26 46
spine02(swp2) 4 65199 2924 2922 0 0 0 02:18:52 26 46
Total number of neighbors 2
cumulus@leaf01:mgmt:~$ net show evpn es
Type: B bypass, L local, R remote, N non-DF
ESI Type ES-IF VTEPs
03:00:00:00:00:00:aa:00:00:01 LR bond1 10.10.10.2
03:00:00:00:00:00:aa:00:00:02 LR bond2 10.10.10.2
cumulus@leaf01:mgmt:~$ net show bgp l2vpn evpn es
ES Flags: B - bypass, L local, R remote, I inconsistent
VTEP Flags: E ESR/Type-4, A active nexthop
ESI Flags RD #VNIs VTEPs
03:00:00:00:00:00:aa:00:00:01 LR 10.10.10.1:3 1 10.10.10.2(EA)
03:00:00:00:00:00:aa:00:00:02 LR 10.10.10.1:4 1 10.10.10.2(EA)
cumulus@leaf01:mgmt:~$ net show bgp l2vpn evpn es-evi
Flags: L local, R remote, I inconsistent
VTEP-Flags: E EAD-per-ES, V EAD-per-EVI
VNI ESI Flags VTEPs
20 03:00:00:00:00:00:aa:00:00:02 LR 10.10.10.2(EV)
10 03:00:00:00:00:00:aa:00:00:01 LR 10.10.10.2(EV)
cumulus@leaf01:mgmt:~$ net show bgp l2vpn evpn es-vrf
ES-VRF Flags: A Active
ESI VRF Flags IPv4-NHG IPv6-NHG Ref
03:00:00:00:00:00:aa:00:00:01 VRF RED A 72580645 72580646 1
03:00:00:00:00:00:aa:00:00:02 VRF GREEN A 72580647 72580648 1
cumulus@leaf01:mgmt:~$ net show int bond1
Name MAC Speed MTU Mode
-- ----- ----------------- ----- ---- -------
UP bond1 48:b0:2d:3d:e9:84 1G 9216 802.3ad
Bond Details
------------------ --------
Bond Mode: 802.3ad
Load Balancing: layer3+4
Minimum Links: 1
LACP Sys Priority:
LACP Rate: 1
LACP Bypass: Active
All VLANs on L2 Port
--------------------
10
Untagged
--------
10
cl-netstat counters
-------------------
RX_OK RX_ERR RX_DRP RX_OVR TX_OK TX_ERR TX_DRP TX_OVR
------ ------ ------ ------ ------ ------ ------ ------
257688 0 1 0 163218 0 0 0
LLDP Details
------------
LocalPort RemotePort(RemoteHost)
--------- ---------------------------
swp3 48:b0:2d:33:24:a3(server01)
Routing
-------
Interface bond1 is up, line protocol is up
Link ups: 4 last: 2023/04/14 15:51:23.59
Link downs: 0 last: (never)
PTM status: disabled
vrf: default
index 10 metric 0 mtu 9216 speed 4294967295
flags: <UP,BROADCAST,RUNNING,MULTICAST>
Type: Ethernet
HWaddr: 48:b0:2d:3d:e9:84
Interface Type bond
Master interface: br_default PVID: 10
EVPN-MH: ES id 1 ES sysmac 00:00:00:00:00:aa
protodown: off (n/a)
ARP-ND redirect enabled: ARP 1713 ND 2791
cumulus@leaf01:mgmt:~$ net show bgp vrf RED
show bgp vrf RED ipv4 unicast
=============================
BGP table version is 57, local router ID is 10.10.10.1, vrf id 19
Default local pref 100, local AS 65101
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
* 192.168.1.0/24 10.10.10.2< 0 65199 65102 ?
* 10.10.10.2< 0 65199 65102 ?
*> 0.0.0.0 0 32768 ?
* 192.168.10.0/24 10.10.20.2< 0 65199 65110 65210 65299 65202 ?
*= 10.10.20.2< 0 65199 65110 65210 65299 65202 ?
*> 10.10.20.1< 0 65199 65110 65210 65299 65201 ?
* 10.10.20.1< 0 65199 65110 65210 65299 65201 ?
Displayed 2 routes and 7 total paths
show bgp vrf RED ipv6 unicast
=============================
No BGP prefixes displayed, 0 exist
cumulus@leaf01:mgmt:~$ net show route vrf RED
show ip route vrf RED
======================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric, Z - FRR,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF RED:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 2d22h09m
C * 192.168.1.0/24 [0/1024] is directly connected, vlan10-v0, 2d19h15m
C>* 192.168.1.0/24 is directly connected, vlan10, 2d22h09m
B>* 192.168.10.0/24 [20/0] via 10.10.20.1, vxlan99 (vrf default) onlink, label 5001, weight 1, 00:02:56
* via 10.10.20.2, vxlan99 (vrf default) onlink, label 5001, weight 1, 00:02:56
show ipv6 route vrf RED
========================
Codes: K - kernel route, C - connected, S - static, R - RIPng,
O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table,
v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR,
f - OpenFabric, Z - FRR,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF RED:
K>* ::/0 [255/8192] unreachable (ICMP unreachable), 2d22h09m
C * fe80::/64 is directly connected, vlan10-v0, 2d19h15m
C * fe80::/64 is directly connected, vlan220_l3, 2d22h08m
C>* fe80::/64 is directly connected, vlan10, 2d22h09m
cumulus@leaf01:mgmt:~$ net show bgp vrf GREEN
show bgp vrf GREEN ipv4 unicast
===============================
BGP table version is 67, local router ID is 10.10.10.1, vrf id 13
Default local pref 100, local AS 65101
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
* 192.168.2.0/24 10.10.10.2< 0 65199 65102 ?
* 10.10.10.2< 0 65199 65102 ?
*> 0.0.0.0 0 32768 ?
* 192.168.20.0/24 10.10.20.2< 0 65199 65110 65210 65299 65202 ?
*= 10.10.20.2< 0 65199 65110 65210 65299 65202 ?
*> 10.10.20.1< 0 65199 65110 65210 65299 65201 ?
* 10.10.20.1< 0 65199 65110 65210 65299 65201 ?
Displayed 2 routes and 7 total paths
show bgp vrf GREEN ipv6 unicast
===============================
No BGP prefixes displayed, 0 exist
cumulus@leaf01:mgmt:~$ net show route vrf GREEN
show ip route vrf GREEN
========================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric, Z - FRR,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF GREEN:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 2d22h09m
C * 192.168.2.0/24 [0/1024] is directly connected, vlan20-v0, 2d19h16m
C>* 192.168.2.0/24 is directly connected, vlan20, 2d22h09m
B>* 192.168.20.0/24 [20/0] via 10.10.20.1, vxlan99 (vrf default) onlink, label 5002, weight 1, 00:03:43
* via 10.10.20.2, vxlan99 (vrf default) onlink, label 5002, weight 1, 00:03:43
show ipv6 route vrf GREEN
==========================
Codes: K - kernel route, C - connected, S - static, R - RIPng,
O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table,
v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR,
f - OpenFabric, Z - FRR,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF GREEN:
K>* ::/0 [255/8192] unreachable (ICMP unreachable), 2d22h09m
C * fe80::/64 is directly connected, vlan20-v0, 2d19h16m
C * fe80::/64 is directly connected, vlan20, 2d22h09m
C>* fe80::/64 is directly connected, vlan370_l3, 2d22h09m
cumulus@leaf01:mgmt:~$ net show bgp vrf RED 192.168.1.0/24
BGP routing table entry for 192.168.1.0/24
Paths: (3 available, best #3, vrf RED)
Not advertised to any peer
Imported from 10.10.10.2:4:[5]:[0]:[24]:[192.168.1.0], VNI 4001
65199 65102
10.10.10.2 from spine01(swp1) (10.10.10.101) announce-nh-self
Origin incomplete, valid, external, bestpath-from-AS 65199
Extended Community: RT:65102:4001 ET:8 Rmac:44:38:39:22:bb:07
Last update: Mon Apr 17 08:46:49 2023
Imported from 10.10.10.2:4:[5]:[0]:[24]:[192.168.1.0], VNI 4001
65199 65102
10.10.10.2 from spine02(swp2) (10.10.10.102) announce-nh-self
Origin incomplete, valid, external
Extended Community: RT:65102:4001 ET:8 Rmac:44:38:39:22:bb:07
Last update: Mon Apr 17 08:46:49 2023
Local
0.0.0.0 from 0.0.0.0 (10.10.10.1)
Origin incomplete, metric 0, weight 32768, valid, sourced, bestpath-from-AS Local, best (Weight)
Last update: Mon Apr 17 08:46:47 2023
cumulus@leaf01:mgmt:~$ net show bgp vrf RED 192.168.10.0/24
BGP routing table entry for 192.168.10.0/24
Paths: (4 available, best #3, vrf RED)
Not advertised to any peer
Imported from 10.10.20.2:6:[5]:[0]:[24]:[192.168.10.0], VNI 5001
65199 65110 65210 65299 65202
10.10.20.2 from spine01(swp1) (10.10.10.101) announce-nh-self
Origin incomplete, valid, external
Extended Community: RT:65202:5001 ET:8 Rmac:44:38:39:22:bb:09
Last update: Mon Apr 17 11:04:25 2023
Imported from 10.10.20.2:6:[5]:[0]:[24]:[192.168.10.0], VNI 5001
65199 65110 65210 65299 65202
10.10.20.2 from spine02(swp2) (10.10.10.102) announce-nh-self
Origin incomplete, valid, external, multipath
Extended Community: RT:65202:5001 ET:8 Rmac:44:38:39:22:bb:09
Last update: Mon Apr 17 11:04:25 2023
Imported from 10.10.20.1:8:[5]:[0]:[24]:[192.168.10.0], VNI 5001
65199 65110 65210 65299 65201
10.10.20.1 from spine01(swp1) (10.10.10.101) announce-nh-self
Origin incomplete, valid, external, multipath, bestpath-from-AS 65199, best (Router ID)
Extended Community: RT:65201:5001 ET:8 Rmac:44:38:39:22:bb:08
Last update: Mon Apr 17 11:04:25 2023
Imported from 10.10.20.1:8:[5]:[0]:[24]:[192.168.10.0], VNI 5001
65199 65110 65210 65299 65201
10.10.20.1 from spine02(swp2) (10.10.10.102) announce-nh-self
Origin incomplete, valid, external
Extended Community: RT:65201:5001 ET:8 Rmac:44:38:39:22:bb:08
Last update: Mon Apr 17 11:04:25 2023
cumulus@leaf01:mgmt:~$ net show bgp evpn vni
Advertise Gateway Macip: Disabled
Advertise SVI Macip: Disabled
Advertise All VNI flag: Enabled
BUM flooding: Head-end replication
VXLAN flooding: Enabled
Number of L2 VNIs: 2
Number of L3 VNIs: 2
Flags: * - Kernel
VNI Type RD Import RT Export RT Tenant VRF
* 20 L2 10.10.10.1:2 65101:20 65101:20 GREEN
* 10 L2 10.10.10.1:7 65101:10 65101:10 RED
* 4002 L3 10.10.10.1:5 0:4002, ... 65101:4002 GREEN
* 4001 L3 10.10.10.1:6 0:4001, ... 65101:4001 RED
cumulus@leaf01:mgmt:~$ net show bgp evpn vni 4001
VNI: 4001 (known to the kernel)
Type: L3
Tenant VRF: RED
RD: 10.10.10.1:6
Originator IP: 10.10.10.1
Advertise-gw-macip : n/a
Advertise-svi-macip : n/a
Advertise-pip: Yes
System-IP: 10.10.10.1
System-MAC: 44:38:39:22:bb:06
Router-MAC: 44:38:39:22:bb:06
Import Route Target:
0:4001
0:5001
Export Route Target:
65101:4001
cumulus@leaf01:mgmt:~$ net show bgp evpn vni 4002
VNI: 4002 (known to the kernel)
Type: L3
Tenant VRF: GREEN
RD: 10.10.10.1:5
Originator IP: 10.10.10.1
Advertise-gw-macip : n/a
Advertise-svi-macip : n/a
Advertise-pip: Yes
System-IP: 10.10.10.1
System-MAC: 44:38:39:22:bb:06
Router-MAC: 44:38:39:22:bb:06
Import Route Target:
0:4002
0:5002
Export Route Target:
65101:4002
cumulus@leaf01:mgmt:~$ net show evpn mac vni all
VNI 10 #MACs (local and remote) 5
Flags: B=bypass N=sync-neighs, I=local-inactive, P=peer-active, X=peer-proxy
MAC Type Flags Intf/Remote ES/VTEP VLAN Seq #'s
44:38:39:22:bb:06 local vlan10 0/0
48:b0:2d:33:24:a3 local X bond1 10 0/0
44:38:39:22:bb:07 remote 10.10.10.2 0/0
42:20:47:91:95:a7 local NP bond1 10 1/0
48:b0:2d:bc:7e:83 remote 10.10.10.2 0/0
VNI 20 #MACs (local and remote) 5
Flags: B=bypass N=sync-neighs, I=local-inactive, P=peer-active, X=peer-proxy
MAC Type Flags Intf/Remote ES/VTEP VLAN Seq #'s
48:b0:2d:9d:31:ae local X bond2 20 0/0
44:38:39:22:bb:06 local vlan20 0/0
48:b0:2d:7f:a9:bd remote 10.10.10.2 0/0
44:38:39:22:bb:07 remote 10.10.10.2 0/0
a6:e0:55:25:f3:b2 local NP bond2 20 1/0
cumulus@leaf02:mgmt:~$ net show bgp sum
show bgp ipv4 unicast summary
=============================
BGP router identifier 10.10.10.2, local AS number 65102 vrf-id 0
BGP table version 205
RIB entries 23, using 4600 bytes of memory
Peers 2, using 46 KiB of memory
Peer groups 1, using 64 bytes of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
spine01(swp1) 4 65199 3015 3013 0 0 0 02:23:25 10 12
spine02(swp2) 4 65199 3015 3013 0 0 0 02:23:25 10 12
Total number of neighbors 2
show bgp ipv6 unicast summary
=============================
% No BGP neighbors found
show bgp l2vpn evpn summary
===========================
BGP router identifier 10.10.10.2, local AS number 65102 vrf-id 0
BGP table version 0
RIB entries 75, using 15 KiB of memory
Peers 2, using 46 KiB of memory
Peer groups 1, using 64 bytes of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
spine01(swp1) 4 65199 3015 3013 0 0 0 02:23:26 24 46
spine02(swp2) 4 65199 3015 3013 0 0 0 02:23:26 24 46
Total number of neighbors 2
cumulus@leaf02:mgmt:~$ net show bgp vrf RED
show bgp vrf RED ipv4 unicast
=============================
BGP table version is 53, local router ID is 10.10.10.2, vrf id 19
Default local pref 100, local AS 65102
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
* 192.168.1.0/24 10.10.10.1< 0 65199 65101 ?
* 10.10.10.1< 0 65199 65101 ?
*> 0.0.0.0 0 32768 ?
* 192.168.10.0/24 10.10.20.2< 0 65199 65110 65210 65299 65202 ?
*= 10.10.20.2< 0 65199 65110 65210 65299 65202 ?
* 10.10.20.1< 0 65199 65110 65210 65299 65201 ?
*> 10.10.20.1< 0 65199 65110 65210 65299 65201 ?
Displayed 2 routes and 7 total paths
show bgp vrf RED ipv6 unicast
=============================
No BGP prefixes displayed, 0 exist
cumulus@leaf02:mgmt:~$ net show bgp vrf GREEN
show bgp vrf GREEN ipv4 unicast
===============================
BGP table version is 66, local router ID is 10.10.10.2, vrf id 13
Default local pref 100, local AS 65102
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
* 192.168.2.0/24 10.10.10.1< 0 65199 65101 ?
* 10.10.10.1< 0 65199 65101 ?
*> 0.0.0.0 0 32768 ?
* 192.168.20.0/24 10.10.20.2< 0 65199 65110 65210 65299 65202 ?
*= 10.10.20.2< 0 65199 65110 65210 65299 65202 ?
* 10.10.20.1< 0 65199 65110 65210 65299 65201 ?
*> 10.10.20.1< 0 65199 65110 65210 65299 65201 ?
Displayed 2 routes and 7 total paths
show bgp vrf GREEN ipv6 unicast
===============================
No BGP prefixes displayed, 0 exist
cumulus@leaf02:mgmt:~$ net show route vrf RED
show ip route vrf RED
======================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric, Z - FRR,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF RED:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 2d22h13m
C * 192.168.1.0/24 [0/1024] is directly connected, vlan10-v0, 2d19h20m
C>* 192.168.1.0/24 is directly connected, vlan10, 2d22h13m
B>* 192.168.10.0/24 [20/0] via 10.10.20.1, vxlan99 (vrf default) onlink, label 5001, weight 1, 00:07:19
* via 10.10.20.2, vxlan99 (vrf default) onlink, label 5001, weight 1, 00:07:19
show ipv6 route vrf RED
========================
Codes: K - kernel route, C - connected, S - static, R - RIPng,
O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table,
v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR,
f - OpenFabric, Z - FRR,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF RED:
K>* ::/0 [255/8192] unreachable (ICMP unreachable), 2d22h13m
C * fe80::/64 is directly connected, vlan10-v0, 2d19h20m
C * fe80::/64 is directly connected, vlan10, 2d22h13m
C>* fe80::/64 is directly connected, vlan220_l3, 2d22h13m
cumulus@leaf02:mgmt:~$ net show route vrf GREEN
show ip route vrf GREEN
========================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric, Z - FRR,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF GREEN:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 2d22h13m
C * 192.168.2.0/24 [0/1024] is directly connected, vlan20-v0, 2d19h20m
C>* 192.168.2.0/24 is directly connected, vlan20, 2d22h13m
B>* 192.168.20.0/24 [20/0] via 10.10.20.1, vxlan99 (vrf default) onlink, label 5002, weight 1, 00:07:23
* via 10.10.20.2, vxlan99 (vrf default) onlink, label 5002, weight 1, 00:07:23
show ipv6 route vrf GREEN
==========================
Codes: K - kernel route, C - connected, S - static, R - RIPng,
O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table,
v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR,
f - OpenFabric, Z - FRR,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF GREEN:
K>* ::/0 [255/8192] unreachable (ICMP unreachable), 2d22h13m
C * fe80::/64 is directly connected, vlan20-v0, 2d19h20m
C * fe80::/64 is directly connected, vlan20, 2d22h13m
C>* fe80::/64 is directly connected, vlan370_l3, 2d22h13m
cumulus@leaf02:mgmt:~$ net show evpn es
Type: B bypass, L local, R remote, N non-DF
ESI Type ES-IF VTEPs
03:00:00:00:00:00:aa:00:00:01 LRN bond1 10.10.10.1
03:00:00:00:00:00:aa:00:00:02 LRN bond2 10.10.10.1
cumulus@leaf02:mgmt:~$ net show bgp l2vpn evpn es
ES Flags: B - bypass, L local, R remote, I inconsistent
VTEP Flags: E ESR/Type-4, A active nexthop
ESI Flags RD #VNIs VTEPs
03:00:00:00:00:00:aa:00:00:01 LR 10.10.10.2:6 1 10.10.10.1(EA)
03:00:00:00:00:00:aa:00:00:02 LR 10.10.10.2:7 1 10.10.10.1(EA)
cumulus@leaf02:mgmt:~$ net show bgp l2vpn evpn es-evi
Flags: L local, R remote, I inconsistent
VTEP-Flags: E EAD-per-ES, V EAD-per-EVI
VNI ESI Flags VTEPs
20 03:00:00:00:00:00:aa:00:00:02 LR 10.10.10.1(EV)
10 03:00:00:00:00:00:aa:00:00:01 LR 10.10.10.1(EV)
cumulus@leaf02:mgmt:~$ net show bgp l2vpn evpn es-vrf
ES-VRF Flags: A Active
ESI VRF Flags IPv4-NHG IPv6-NHG Ref
03:00:00:00:00:00:aa:00:00:01 VRF RED A 72580647 72580648 1
03:00:00:00:00:00:aa:00:00:02 VRF GREEN A 72580645 72580646 1
cumulus@leaf02:mgmt:~$ net show bgp evpn vni
Advertise Gateway Macip: Disabled
Advertise SVI Macip: Disabled
Advertise All VNI flag: Enabled
BUM flooding: Head-end replication
VXLAN flooding: Enabled
Number of L2 VNIs: 2
Number of L3 VNIs: 2
Flags: * - Kernel
VNI Type RD Import RT Export RT Tenant VRF
* 20 L2 10.10.10.2:2 65102:20 65102:20 GREEN
* 10 L2 10.10.10.2:3 65102:10 65102:10 RED
* 4001 L3 10.10.10.2:4 0:4001, ... 65102:4001 RED
* 4002 L3 10.10.10.2:5 0:4002, ... 65102:4002 GREEN
cumulus@leaf02:mgmt:~$ net show int bond1
Name MAC Speed MTU Mode
-- ----- ----------------- ----- ---- -------
UP bond1 48:b0:2d:f0:64:6e 1G 9216 802.3ad
Bond Details
------------------ --------
Bond Mode: 802.3ad
Load Balancing: layer3+4
Minimum Links: 1
LACP Sys Priority:
LACP Rate: 1
LACP Bypass: Active
All VLANs on L2 Port
--------------------
10
Untagged
--------
10
cl-netstat counters
-------------------
RX_OK RX_ERR RX_DRP RX_OVR TX_OK TX_ERR TX_DRP TX_OVR
------ ------ ------ ------ ------ ------ ------ ------
255593 0 0 0 165821 0 0 0
LLDP Details
------------
LocalPort RemotePort(RemoteHost)
--------- ---------------------------
swp3 48:b0:2d:bc:7e:83(server01)
Routing
-------
Interface bond1 is up, line protocol is up
Link ups: 6 last: 2023/04/14 15:51:19.73
Link downs: 0 last: (never)
PTM status: disabled
vrf: default
index 9 metric 0 mtu 9216 speed 1000
flags: <UP,BROADCAST,RUNNING,MULTICAST>
Type: Ethernet
HWaddr: 48:b0:2d:f0:64:6e
Interface Type bond
Master interface: br_default PVID: 10
EVPN-MH: ES id 1 ES sysmac 00:00:00:00:00:aa
protodown: off (n/a)
ARP-ND redirect enabled: ARP 906 ND 3030
cumulus@leaf02:mgmt:~$ net show bgp evpn vni 4001
VNI: 4001 (known to the kernel)
Type: L3
Tenant VRF: RED
RD: 10.10.10.2:4
Originator IP: 10.10.10.2
Advertise-gw-macip : n/a
Advertise-svi-macip : n/a
Advertise-pip: Yes
System-IP: 10.10.10.2
System-MAC: 44:38:39:22:bb:07
Router-MAC: 44:38:39:22:bb:07
Import Route Target:
0:4001
0:5001
Export Route Target:
65102:4001
cumulus@leaf02:mgmt:~$ net show bgp evpn vni 4002
VNI: 4002 (known to the kernel)
Type: L3
Tenant VRF: GREEN
RD: 10.10.10.2:5
Originator IP: 10.10.10.2
Advertise-gw-macip : n/a
Advertise-svi-macip : n/a
Advertise-pip: Yes
System-IP: 10.10.10.2
System-MAC: 44:38:39:22:bb:07
Router-MAC: 44:38:39:22:bb:07
Import Route Target:
0:4002
0:5002
Export Route Target:
65102:4002
cumulus@leaf02:mgmt:~$ net show evpn mac vni all
VNI 10 #MACs (local and remote) 5
Flags: B=bypass N=sync-neighs, I=local-inactive, P=peer-active, X=peer-proxy
MAC Type Flags Intf/Remote ES/VTEP VLAN Seq #'s
44:38:39:22:bb:06 remote 10.10.10.1 0/0
48:b0:2d:33:24:a3 local PI bond1 10 0/0
44:38:39:22:bb:07 local vlan10 0/0
42:20:47:91:95:a7 local NP bond1 10 1/0
48:b0:2d:bc:7e:83 local bond1 10 0/0
VNI 20 #MACs (local and remote) 5
Flags: B=bypass N=sync-neighs, I=local-inactive, P=peer-active, X=peer-proxy
MAC Type Flags Intf/Remote ES/VTEP VLAN Seq #'s
48:b0:2d:9d:31:ae local PI bond2 20 0/0
44:38:39:22:bb:06 remote 10.10.10.1 0/0
48:b0:2d:7f:a9:bd local bond2 20 0/0
44:38:39:22:bb:07 local vlan20 0/0
a6:e0:55:25:f3:b2 local NP bond2 20 1/0
验证网桥 `br_default` 正在学习 MAC 条目
cumulus@leaf01:mgmt:~$ nv show bridge domain br_default mac-table
age bridge-domain entry-type interface last-update MAC address src-vni vlan vni Summary
-- ------ ------------- ---------- ---------- ----------- ----------------- ------- ---- ---- ----------------------
0 3 br_default static bond2 9770 48:b0:2d:9d:31:ae 20
1 58 br_default static bond2 237427 a6:e0:55:25:f3:b2 20
2 253880 br_default permanent bond2 253880 48:b0:2d:ce:f9:6f
3 9770 br_default vxlan48 9770 48:b0:2d:bc:7e:83 10 None remote-dst: 10.10.10.2
4 9770 br_default vxlan48 9770 44:38:39:22:bb:07 10 None remote-dst: 10.10.10.2
5 9770 br_default vxlan48 9770 48:b0:2d:7f:a9:bd 20 None remote-dst: 10.10.10.2
6 253880 br_default permanent vxlan48 253880 96:8b:4a:de:a2:71 None
7 9770 permanent vxlan48 4 00:00:00:00:00:00 10 None remote-dst: 10.10.10.2
8 26 br_default static bond1 9770 48:b0:2d:33:24:a3 10
9 53 br_default static bond1 235497 42:20:47:91:95:a7 10
10 253880 br_default permanent bond1 253880 48:b0:2d:3d:e9:84
11 permanent br_default 00:00:5e:00:01:0a
12 243496 br_default permanent br_default 243496 44:38:39:22:bb:06 10
cumulus@leaf02:mgmt:~$ nv show bridge domain br_default mac-table
age bridge-domain entry-type interface last-update MAC address src-vni vlan vni Summary
-- ------ ------------- ---------- ---------- ----------- ----------------- ------- ---- ---- ----------------------
0 250398 br_default static bond2 250398 48:b0:2d:9d:31:ae 20
1 65 br_default static bond2 250384 a6:e0:55:25:f3:b2 20
2 2 br_default bond2 253825 48:b0:2d:7f:a9:bd 20
3 253891 br_default permanent bond2 253891 48:b0:2d:aa:08:bd
4 250398 br_default static bond1 250398 48:b0:2d:33:24:a3 10
5 105 br_default static bond1 235506 42:20:47:91:95:a7 10
6 2 br_default bond1 253825 48:b0:2d:bc:7e:83 10
7 253891 br_default permanent bond1 253891 48:b0:2d:f0:64:6e
8 9775 br_default vxlan48 65 44:38:39:22:bb:06 20 None remote-dst: 10.10.10.1
9 253891 br_default permanent vxlan48 253891 1a:64:66:e8:14:39 1 None
10 9775 permanent vxlan48 50 00:00:00:00:00:00 10 None remote-dst: 10.10.10.1
11 permanent br_default 00:00:5e:00:01:0a
12 243505 br_default permanent br_default 243505 44:38:39:22:bb:07 10
从上表中，找到三层 VLAN 接口 MAC 地址和 VRR MAC 地址
cumulus@leaf01:mgmt:~$ nv show int vlan10 ip vrr
operational applied
----------- ----------------- --------------
enable on
mac-address 00:00:5e:00:01:0a auto
mac-id none
[address] 192.168.1.1/24 192.168.1.1/24
state up up
cumulus@leaf01:mgmt:~$ nv show int vlan10 | grep mac
mac-address auto
mac-id none
mac 44:38:39:22:bb:06
cumulus@leaf02:mgmt:~$ nv show int vlan10 | grep mac
mac-address auto
mac-id none
mac 44:38:39:22:bb:07
验证入口 PE (leaf01) 上终端主机 *192.168.10.110* 的 EVPN type-5 路由，该主机连接到 leaf03 和 leaf04
cumulus@leaf01:mgmt:~$ net show bgp evpn route type 5 | grep 192.168.10 -A 4 -B 1
Route Distinguisher: 10.10.20.1:8
* [5]:[0]:[24]:[192.168.10.0] RD 10.10.20.1:8
10.10.20.1 (spine02)
0 65199 65110 65210 65299 65201 ?
RT:65201:5001 ET:8 Rmac:44:38:39:22:bb:08
*> [5]:[0]:[24]:[192.168.10.0] RD 10.10.20.1:8
10.10.20.1 (spine01)
0 65199 65110 65210 65299 65201 ?
RT:65201:5001 ET:8 Rmac:44:38:39:22:bb:08
Route Distinguisher: 10.10.20.2:6
* [5]:[0]:[24]:[192.168.10.0] RD 10.10.20.2:6
10.10.20.2 (spine02)
0 65199 65110 65210 65299 65202 ?
RT:65202:5001 ET:8 Rmac:44:38:39:22:bb:09
*> [5]:[0]:[24]:[192.168.10.0] RD 10.10.20.2:6
10.10.20.2 (spine01)
0 65199 65110 65210 65299 65202 ?
RT:65202:5001 ET:8 Rmac:44:38:39:22:bb:09
Route Distinguisher: 10.10.20.2:7
验证出口 PE (leaf03) 上终端主机 192.168.10.110 的 EVPN type-5 路由，该主机连接到 leaf03 和 leaf04
cumulus@leaf03:mgmt:~$ net show bgp evpn route type 5 | grep 192.168.10 -A 4 -B 1
Route Distinguisher: 10.10.20.1:8
*> [5]:[0]:[24]:[192.168.10.0] RD 10.10.20.1:8
10.10.20.1 (leaf03)
0 32768 ?
ET:8 RT:65201:5001 Rmac:44:38:39:22:bb:08
Route Distinguisher: 10.10.20.2:6
* [5]:[0]:[24]:[192.168.10.0] RD 10.10.20.2:6
10.10.20.2 (spine03)
0 65299 65202 ?
RT:65202:5001 ET:8 Rmac:44:38:39:22:bb:09
*> [5]:[0]:[24]:[192.168.10.0] RD 10.10.20.2:6
10.10.20.2 (spine04)
0 65299 65202 ?
RT:65202:5001 ET:8 Rmac:44:38:39:22:bb:09
Route Distinguisher: 10.10.20.2:7
验证边界叶节点上的路由
cumulus@borderleaf01:mgmt:~$ net show bgp sum
show bgp ipv4 unicast summary
=============================
BGP router identifier 10.10.10.10, local AS number 65110 vrf-id 0
BGP table version 12
RIB entries 19, using 3800 bytes of memory
Peers 3, using 68 KiB of memory
Peer groups 2, using 128 bytes of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
borderleaf04(swp3) 4 65210 735 734 0 0 0 00:35:39 5 10
spine01(swp1) 4 65199 778 776 0 0 0 00:36:20 3 10
spine02(swp2) 4 65199 777 775 0 0 0 00:36:16 3 10
Total number of neighbors 3
show bgp ipv6 unicast summary
=============================
% No BGP neighbors found
show bgp l2vpn evpn summary
===========================
BGP router identifier 10.10.10.10, local AS number 65110 vrf-id 0
BGP table version 0
RIB entries 31, using 6200 bytes of memory
Peers 3, using 68 KiB of memory
Peer groups 2, using 128 bytes of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
borderleaf04(swp3) 4 65210 735 734 0 0 0 00:35:39 4 8
spine01(swp1) 4 65199 778 776 0 0 0 00:36:20 42 46
spine02(swp2) 4 65199 777 775 0 0 0 00:36:16 42 46
Total number of neighbors 3
cumulus@borderleaf01:mgmt:~$ net show bgp vrf RED
show bgp vrf RED ipv4 unicast
=============================
BGP table version is 3, local router ID is 10.10.10.10, vrf id 13
Default local pref 100, local AS 65110
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
* 192.168.1.0/24 10.10.10.2< 0 65199 65102 ?
* 10.10.10.1< 0 65199 65101 ?
*= 10.10.10.2< 0 65199 65102 ?
*> 10.10.10.1< 0 65199 65101 ?
* 192.168.1.10/32 10.10.10.2< 0 65199 65102 i
* 10.10.10.1< 0 65199 65101 i
*= 10.10.10.2< 0 65199 65102 i
*> 10.10.10.1< 0 65199 65101 i
*= 192.168.10.0/24 10.10.20.2< 0 65210 65299 65202 ?
*> 10.10.20.1< 0 65210 65299 65201 ?
Displayed 3 routes and 10 total paths
show bgp vrf RED ipv6 unicast
=============================
No BGP prefixes displayed, 0 exist
cumulus@borderleaf01:mgmt:~$ net show bgp evpn route type 5 | grep 192.168.1 -A 3 -B 1
Route Distinguisher: 10.10.10.1:6
* [5]:[0]:[24]:[192.168.1.0] RD 10.10.10.1:6
10.10.10.1 (spine02)
0 65199 65101 ?
RT:65101:4001 ET:8 Rmac:44:38:39:22:bb:06
*> [5]:[0]:[24]:[192.168.1.0] RD 10.10.10.1:6
10.10.10.1 (spine01)
0 65199 65101 ?
RT:65101:4001 ET:8 Rmac:44:38:39:22:bb:06
Route Distinguisher: 10.10.10.2:4
* [5]:[0]:[24]:[192.168.1.0] RD 10.10.10.2:4
10.10.10.2 (spine02)
0 65199 65102 ?
RT:65102:4001 ET:8 Rmac:44:38:39:22:bb:07
*> [5]:[0]:[24]:[192.168.1.0] RD 10.10.10.2:4
10.10.10.2 (spine01)
0 65199 65102 ?
RT:65102:4001 ET:8 Rmac:44:38:39:22:bb:07
--
Route Distinguisher: 10.10.20.1:8
*> [5]:[0]:[24]:[192.168.10.0] RD 10.10.20.1:8
10.10.20.1 (borderleaf04)
0 65210 65299 65201 ?
RT:65201:5001 ET:8 Rmac:44:38:39:22:bb:08
Route Distinguisher: 10.10.20.2:6
*> [5]:[0]:[24]:[192.168.10.0] RD 10.10.20.2:6
10.10.20.2 (borderleaf04)
0 65210 65299 65202 ?
RT:65202:5001 ET:8 Rmac:44:38:39:22:bb:09
cumulus@leaf03:mgmt:~$ net show bgp sum
show bgp ipv4 unicast summary
=============================
BGP router identifier 10.10.20.1, local AS number 65201 vrf-id 0
BGP table version 57
RIB entries 23, using 4600 bytes of memory
Peers 2, using 46 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
spine03(swp1) 4 65299 79699 79701 0 0 0 2d18h00m 10 12
spine04(swp2) 4 65299 79714 79714 0 0 0 2d18h00m 10 12
Total number of neighbors 2
show bgp ipv6 unicast summary
=============================
% No BGP neighbors found
show bgp l2vpn evpn summary
===========================
BGP router identifier 10.10.20.1, local AS number 65201 vrf-id 0
BGP table version 0
RIB entries 71, using 14 KiB of memory
Peers 2, using 46 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
spine03(swp1) 4 65299 79699 79702 0 0 0 2d18h00m 25 46
spine04(swp2) 4 65299 79714 79715 0 0 0 2d18h00m 25 46
Total number of neighbors 2
cumulus@leaf03:mgmt:~$ net show evpn es
Type: B bypass, L local, R remote, N non-DF
ESI Type ES-IF VTEPs
03:00:00:00:00:00:bb:00:00:01 LR bond1 10.10.20.2
03:00:00:00:00:00:bb:00:00:02 LR bond2 10.10.20.2
cumulus@leaf03:mgmt:~$ net show bgp l2vpn evpn es
ES Flags: B - bypass, L local, R remote, I inconsistent
VTEP Flags: E ESR/Type-4, A active nexthop
ESI Flags RD #VNIs VTEPs
03:00:00:00:00:00:bb:00:00:01 LR 10.10.20.1:3 1 10.10.20.2(EA)
03:00:00:00:00:00:bb:00:00:02 LR 10.10.20.1:4 1 10.10.20.2(EA)
cumulus@leaf03:mgmt:~$ net show bgp l2vpn evpn es-evi
Flags: L local, R remote, I inconsistent
VTEP-Flags: E EAD-per-ES, V EAD-per-EVI
VNI ESI Flags VTEPs
2020 03:00:00:00:00:00:bb:00:00:02 LR 10.10.20.2(EV)
1010 03:00:00:00:00:00:bb:00:00:01 LR 10.10.20.2(EV)
cumulus@leaf03:mgmt:~$ net show bgp l2vpn evpn es-vrf
ES-VRF Flags: A Active
ESI VRF Flags IPv4-NHG IPv6-NHG Ref
03:00:00:00:00:00:bb:00:00:01 VRF RED A 72580647 72580648 1
03:00:00:00:00:00:bb:00:00:02 VRF GREEN A 72580645 72580646 1
cumulus@leaf03:mgmt:~$ net show int bond1
Name MAC Speed MTU Mode
-- ----- ----------------- ----- ---- -------
UP bond1 48:b0:2d:7d:9c:74 1G 9216 802.3ad
Bond Details
------------------ --------
Bond Mode: 802.3ad
Load Balancing: layer3+4
Minimum Links: 1
LACP Sys Priority:
LACP Rate: 1
LACP Bypass: Active
All VLANs on L2 Port
--------------------
1010
Untagged
--------
1010
cl-netstat counters
-------------------
RX_OK RX_ERR RX_DRP RX_OVR TX_OK TX_ERR TX_DRP TX_OVR
------ ------ ------ ------ ------ ------ ------ ------
241819 0 0 0 157114 0 0 0
LLDP Details
------------
LocalPort RemotePort(RemoteHost)
--------- ---------------------------
swp3 48:b0:2d:72:28:ff(server03)
Routing
-------
Interface bond1 is up, line protocol is up
Link ups: 2 last: 2023/04/17 08:44:08.84
Link downs: 0 last: (never)
PTM status: disabled
vrf: default
index 9 metric 0 mtu 9216 speed 1000
flags: <UP,BROADCAST,RUNNING,MULTICAST>
Type: Ethernet
HWaddr: 48:b0:2d:7d:9c:74
Interface Type bond
Master interface: br_default PVID: 1010
EVPN-MH: ES id 1 ES sysmac 00:00:00:00:00:bb
protodown: off (n/a)
ARP-ND redirect enabled: ARP 1653 ND 2667
cumulus@leaf03:mgmt:~$ net show bgp vrf RED
show bgp vrf RED ipv4 unicast
=============================
BGP table version is 11, local router ID is 10.10.20.1, vrf id 19
Default local pref 100, local AS 65201
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
* 192.168.1.0/24 10.10.10.2< 0 65299 65210 65110 65199 65102 ?
* 10.10.10.1< 0 65299 65210 65110 65199 65101 ?
*= 10.10.10.2< 0 65299 65210 65110 65199 65102 ?
*> 10.10.10.1< 0 65299 65210 65110 65199 65101 ?
* 192.168.10.0/24 10.10.20.2< 0 65299 65202 ?
* 10.10.20.2< 0 65299 65202 ?
*> 0.0.0.0 0 32768 ?
Displayed 2 routes and 7 total paths
show bgp vrf RED ipv6 unicast
=============================
No BGP prefixes displayed, 0 exist
cumulus@leaf03:mgmt:~$ net show bgp vrf GREEN
show bgp vrf GREEN ipv4 unicast
===============================
BGP table version is 10, local router ID is 10.10.20.1, vrf id 13
Default local pref 100, local AS 65201
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
* 192.168.2.0/24 10.10.10.2< 0 65299 65210 65110 65199 65102 ?
* 10.10.10.1< 0 65299 65210 65110 65199 65101 ?
*= 10.10.10.2< 0 65299 65210 65110 65199 65102 ?
*> 10.10.10.1< 0 65299 65210 65110 65199 65101 ?
* 192.168.20.0/24 10.10.20.2< 0 65299 65202 ?
* 10.10.20.2< 0 65299 65202 ?
*> 0.0.0.0 0 32768 ?
Displayed 2 routes and 7 total paths
show bgp vrf GREEN ipv6 unicast
===============================
No BGP prefixes displayed, 0 exist
cumulus@leaf03:mgmt:~$ net show route vrf RED
show ip route vrf RED
======================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric, Z - FRR,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF RED:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 2d18h01m
B>* 192.168.1.0/24 [20/0] via 10.10.10.1, vxlan99 (vrf default) onlink, label 4001, weight 1, 00:38:33
* via 10.10.10.2, vxlan99 (vrf default) onlink, label 4001, weight 1, 00:38:33
C * 192.168.10.0/24 [0/1024] is directly connected, vlan1010-v0, 02:56:10
C>* 192.168.10.0/24 is directly connected, vlan1010, 02:56:10
show ipv6 route vrf RED
========================
Codes: K - kernel route, C - connected, S - static, R - RIPng,
O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table,
v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR,
f - OpenFabric, Z - FRR,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF RED:
K>* ::/0 [255/8192] unreachable (ICMP unreachable), 2d18h01m
C * fe80::/64 is directly connected, vlan1010, 02:56:08
C * fe80::/64 is directly connected, vlan1010-v0, 02:56:10
C>* fe80::/64 is directly connected, vlan220_l3, 2d18h01m
cumulus@leaf03:mgmt:~$ net show route vrf GREEN
show ip route vrf GREEN
========================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric, Z - FRR,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF GREEN:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 2d18h01m
B>* 192.168.2.0/24 [20/0] via 10.10.10.1, vxlan99 (vrf default) onlink, label 4002, weight 1, 00:38:36
* via 10.10.10.2, vxlan99 (vrf default) onlink, label 4002, weight 1, 00:38:36
C * 192.168.20.0/24 [0/1024] is directly connected, vlan2020-v0, 02:56:13
C>* 192.168.20.0/24 is directly connected, vlan2020, 02:56:13
show ipv6 route vrf GREEN
==========================
Codes: K - kernel route, C - connected, S - static, R - RIPng,
O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table,
v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR,
f - OpenFabric, Z - FRR,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF GREEN:
K>* ::/0 [255/8192] unreachable (ICMP unreachable), 2d18h01m
C * fe80::/64 is directly connected, vlan2020, 02:56:12
C * fe80::/64 is directly connected, vlan2020-v0, 02:56:14
C>* fe80::/64 is directly connected, vlan370_l3, 2d18h01m
cumulus@leaf03:mgmt:~$ net show bgp vrf RED 192.168.10.0/24
BGP routing table entry for 192.168.10.0/24
Paths: (3 available, best #3, vrf RED)
Not advertised to any peer
Imported from 10.10.20.2:6:[5]:[0]:[24]:[192.168.10.0], VNI 5001
65299 65202
10.10.20.2 from spine03(swp1) (10.10.20.103) announce-nh-self
Origin incomplete, valid, external, bestpath-from-AS 65299
Extended Community: RT:65202:5001 ET:8 Rmac:44:38:39:22:bb:09
Last update: Mon Apr 17 08:46:49 2023
Imported from 10.10.20.2:6:[5]:[0]:[24]:[192.168.10.0], VNI 5001
65299 65202
10.10.20.2 from spine04(swp2) (10.10.20.104) announce-nh-self
Origin incomplete, valid, external
Extended Community: RT:65202:5001 ET:8 Rmac:44:38:39:22:bb:09
Last update: Mon Apr 17 08:46:49 2023
Local
0.0.0.0 from 0.0.0.0 (10.10.20.1)
Origin incomplete, metric 0, weight 32768, valid, sourced, bestpath-from-AS Local, best (Weight)
Last update: Mon Apr 17 08:46:48 2023
cumulus@leaf03:mgmt:~$ net show bgp vrf RED 192.168.1.0/24
BGP routing table entry for 192.168.1.0/24
Paths: (4 available, best #4, vrf RED)
Not advertised to any peer
Imported from 10.10.10.2:4:[5]:[0]:[24]:[192.168.1.0], VNI 4001
65299 65210 65110 65199 65102
10.10.10.2 from spine03(swp1) (10.10.20.103) announce-nh-self
Origin incomplete, valid, external
Extended Community: RT:65102:4001 ET:8 Rmac:44:38:39:22:bb:07
Last update: Mon Apr 17 11:04:26 2023
Imported from 10.10.10.1:6:[5]:[0]:[24]:[192.168.1.0], VNI 4001
65299 65210 65110 65199 65101
10.10.10.1 from spine03(swp1) (10.10.20.103) announce-nh-self
Origin incomplete, valid, external
Extended Community: RT:65101:4001 ET:8 Rmac:44:38:39:22:bb:06
Last update: Mon Apr 17 11:04:26 2023
Imported from 10.10.10.2:4:[5]:[0]:[24]:[192.168.1.0], VNI 4001
65299 65210 65110 65199 65102
10.10.10.2 from spine04(swp2) (10.10.20.104) announce-nh-self
Origin incomplete, valid, external, multipath
Extended Community: RT:65102:4001 ET:8 Rmac:44:38:39:22:bb:07
Last update: Mon Apr 17 11:04:24 2023
Imported from 10.10.10.1:6:[5]:[0]:[24]:[192.168.1.0], VNI 4001
65299 65210 65110 65199 65101
10.10.10.1 from spine04(swp2) (10.10.20.104) announce-nh-self
Origin incomplete, valid, external, multipath, bestpath-from-AS 65299, best (Older Path)
Extended Community: RT:65101:4001 ET:8 Rmac:44:38:39:22:bb:06
Last update: Mon Apr 17 11:04:24 2023
cumulus@leaf03:mgmt:~$ net show bgp evpn vni
Advertise Gateway Macip: Disabled
Advertise SVI Macip: Disabled
Advertise All VNI flag: Enabled
BUM flooding: Head-end replication
VXLAN flooding: Enabled
Number of L2 VNIs: 2
Number of L3 VNIs: 2
Flags: * - Kernel
VNI Type RD Import RT Export RT Tenant VRF
* 2020 L2 10.10.20.1:2 65201:2020 65201:2020 GREEN
* 1010 L2 10.10.20.1:9 65201:1010 65201:1010 RED
* 5002 L3 10.10.20.1:7 0:4002, ... 65201:5002 GREEN
* 5001 L3 10.10.20.1:8 0:4001, ... 65201:5001 RED
cumulus@leaf03:mgmt:~$ net show bgp evpn vni 5001
VNI: 5001 (known to the kernel)
Type: L3
Tenant VRF: RED
RD: 10.10.20.1:8
Originator IP: 10.10.20.1
Advertise-gw-macip : n/a
Advertise-svi-macip : n/a
Advertise-pip: Yes
System-IP: 10.10.20.1
System-MAC: 44:38:39:22:bb:08
Router-MAC: 44:38:39:22:bb:08
Import Route Target:
0:4001
0:5001
Export Route Target:
65201:5001
cumulus@leaf03:mgmt:~$ net show bgp evpn vni 5002
VNI: 5002 (known to the kernel)
Type: L3
Tenant VRF: GREEN
RD: 10.10.20.1:7
Originator IP: 10.10.20.1
Advertise-gw-macip : n/a
Advertise-svi-macip : n/a
Advertise-pip: Yes
System-IP: 10.10.20.1
System-MAC: 44:38:39:22:bb:08
Router-MAC: 44:38:39:22:bb:08
Import Route Target:
0:4002
0:5002
Export Route Target:
65201:5002
cumulus@leaf03:mgmt:~$ net show evpn mac vni all
VNI 1010 #MACs (local and remote) 5
Flags: B=bypass N=sync-neighs, I=local-inactive, P=peer-active, X=peer-proxy
MAC Type Flags Intf/Remote ES/VTEP VLAN Seq #'s
48:b0:2d:72:28:ff local P bond1 1010 0/0
48:b0:2d:82:05:04 local P bond1 1010 0/0
b6:4b:0f:ea:f2:02 local NP bond1 1010 0/0
44:38:39:22:bb:09 remote 10.10.20.2 0/0
44:38:39:22:bb:08 local vlan1010 0/0
VNI 2020 #MACs (local and remote) 5
Flags: B=bypass N=sync-neighs, I=local-inactive, P=peer-active, X=peer-proxy
MAC Type Flags Intf/Remote ES/VTEP VLAN Seq #'s
ee:54:69:be:3a:3f local NP bond2 2020 0/0
48:b0:2d:75:8e:7a local P bond2 2020 0/0
44:38:39:22:bb:09 remote 10.10.20.2 0/0
48:b0:2d:a7:e2:6e local P bond2 2020 0/0
44:38:39:22:bb:08 local vlan2020 0/0
cumulus@leaf04:mgmt:~$ net show bgp sum
show bgp ipv4 unicast summary
=============================
BGP router identifier 10.10.20.2, local AS number 65202 vrf-id 0
BGP table version 21
RIB entries 23, using 4600 bytes of memory
Peers 2, using 46 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
spine03(swp1) 4 65299 229 236 0 0 0 00:08:42 10 12
spine04(swp2) 4 65299 224 231 0 0 0 00:08:34 10 12
Total number of neighbors 2
show bgp ipv6 unicast summary
=============================
% No BGP neighbors found
show bgp l2vpn evpn summary
===========================
BGP router identifier 10.10.20.2, local AS number 65202 vrf-id 0
BGP table version 0
RIB entries 31, using 6200 bytes of memory
Peers 2, using 46 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
spine03(swp1) 4 65299 229 236 0 0 0 00:08:42 25 46
spine04(swp2) 4 65299 224 231 0 0 0 00:08:34 25 46
Total number of neighbors 2
cumulus@leaf04:mgmt:~$ net show bgp vrf RED
show bgp vrf RED ipv4 unicast
=============================
BGP table version is 2, local router ID is 10.10.20.2, vrf id 19
Default local pref 100, local AS 65202
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
* 192.168.1.0/24 10.10.10.2< 0 65299 65210 65110 65199 65102 ?
* 10.10.10.1< 0 65299 65210 65110 65199 65101 ?
*= 10.10.10.2< 0 65299 65210 65110 65199 65102 ?
*> 10.10.10.1< 0 65299 65210 65110 65199 65101 ?
* 192.168.10.0/24 10.10.20.1< 0 65299 65201 ?
* 10.10.20.1< 0 65299 65201 ?
*> 0.0.0.0 0 32768 ?
Displayed 2 routes and 7 total paths
show bgp vrf RED ipv6 unicast
=============================
No BGP prefixes displayed, 0 exist
cumulus@leaf04:mgmt:~$ net show bgp vrf GREEN
show bgp vrf GREEN ipv4 unicast
===============================
BGP table version is 2, local router ID is 10.10.20.2, vrf id 13
Default local pref 100, local AS 65202
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
* 192.168.2.0/24 10.10.10.2< 0 65299 65210 65110 65199 65102 ?
* 10.10.10.1< 0 65299 65210 65110 65199 65101 ?
*= 10.10.10.2< 0 65299 65210 65110 65199 65102 ?
*> 10.10.10.1< 0 65299 65210 65110 65199 65101 ?
* 192.168.20.0/24 10.10.20.1< 0 65299 65201 ?
* 10.10.20.1< 0 65299 65201 ?
*> 0.0.0.0 0 32768 ?
Displayed 2 routes and 7 total paths
show bgp vrf GREEN ipv6 unicast
===============================
No BGP prefixes displayed, 0 exist
cumulus@leaf04:mgmt:~$ net show route vrf RED
show ip route vrf RED
======================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric, Z - FRR,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF RED:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 00:09:06
B>* 192.168.1.0/24 [20/0] via 10.10.10.1, vxlan99 (vrf default) onlink, label 4001, weight 1, 00:09:03
* via 10.10.10.2, vxlan99 (vrf default) onlink, label 4001, weight 1, 00:09:03
C * 192.168.10.0/24 [0/1024] is directly connected, vlan1010-v0, 00:09:06
C>* 192.168.10.0/24 is directly connected, vlan1010, 00:09:06
show ipv6 route vrf RED
========================
Codes: K - kernel route, C - connected, S - static, R - RIPng,
O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table,
v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR,
f - OpenFabric, Z - FRR,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF RED:
K>* ::/0 [255/8192] unreachable (ICMP unreachable), 00:09:06
C * fe80::/64 is directly connected, vlan220_l3, 00:09:05
C * fe80::/64 is directly connected, vlan1010, 00:09:06
C>* fe80::/64 is directly connected, vlan1010-v0, 00:09:06
cumulus@leaf04:mgmt:~$ net show route vrf GREEN
show ip route vrf GREEN
========================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric, Z - FRR,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF GREEN:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 00:09:13
B>* 192.168.2.0/24 [20/0] via 10.10.10.1, vxlan99 (vrf default) onlink, label 4002, weight 1, 00:09:10
* via 10.10.10.2, vxlan99 (vrf default) onlink, label 4002, weight 1, 00:09:10
C * 192.168.20.0/24 [0/1024] is directly connected, vlan2020-v0, 00:09:13
C>* 192.168.20.0/24 is directly connected, vlan2020, 00:09:13
show ipv6 route vrf GREEN
==========================
Codes: K - kernel route, C - connected, S - static, R - RIPng,
O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table,
v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR,
f - OpenFabric, Z - FRR,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF GREEN:
K>* ::/0 [255/8192] unreachable (ICMP unreachable), 00:09:13
C * fe80::/64 is directly connected, vlan370_l3, 00:09:13
C * fe80::/64 is directly connected, vlan2020-v0, 00:09:13
C>* fe80::/64 is directly connected, vlan2020, 00:09:13
cumulus@leaf04:mgmt:~$ net show evpn es
Type: B bypass, L local, R remote, N non-DF
ESI Type ES-IF VTEPs
03:00:00:00:00:00:bb:00:00:01 LRN bond1 10.10.20.1
03:00:00:00:00:00:bb:00:00:02 LRN bond2 10.10.20.1
cumulus@leaf04:mgmt:~$ net show bgp l2vpn evpn es
ES Flags: B - bypass, L local, R remote, I inconsistent
VTEP Flags: E ESR/Type-4, A active nexthop
ESI Flags RD #VNIs VTEPs
03:00:00:00:00:00:bb:00:00:01 LR 10.10.20.2:3 1 10.10.20.1(EA)
03:00:00:00:00:00:bb:00:00:02 LR 10.10.20.2:4 1 10.10.20.1(EA)
cumulus@leaf04:mgmt:~$ net show bgp l2vpn evpn es-evi
Flags: L local, R remote, I inconsistent
VTEP-Flags: E EAD-per-ES, V EAD-per-EVI
VNI ESI Flags VTEPs
2020 03:00:00:00:00:00:bb:00:00:02 LR 10.10.20.1(EV)
1010 03:00:00:00:00:00:bb:00:00:01 LR 10.10.20.1(EV)
cumulus@leaf04:mgmt:~$ net show bgp l2vpn evpn es-vrf
ES-VRF Flags: A Active
ESI VRF Flags IPv4-NHG IPv6-NHG Ref
03:00:00:00:00:00:bb:00:00:01 VRF RED A 72580645 72580646 1
03:00:00:00:00:00:bb:00:00:02 VRF GREEN A 72580647 72580648 1
cumulus@leaf04:mgmt:~$ net show bgp evpn vni
Advertise Gateway Macip: Disabled
Advertise SVI Macip: Disabled
Advertise All VNI flag: Enabled
BUM flooding: Head-end replication
VXLAN flooding: Enabled
Number of L2 VNIs: 2
Number of L3 VNIs: 2
Flags: * - Kernel
VNI Type RD Import RT Export RT Tenant VRF
* 2020 L2 10.10.20.2:2 65202:2020 65202:2020 GREEN
* 1010 L2 10.10.20.2:9 65202:1010 65202:1010 RED
* 5002 L3 10.10.20.2:7 0:4002, ... 65202:5002 GREEN
* 5001 L3 10.10.20.2:8 0:4001, ... 65202:5001 RED
cumulus@leaf04:mgmt:~$ net show int bond1
Name MAC Speed MTU Mode
-- ----- ----------------- ----- ---- -------
UP bond1 48:b0:2d:84:e8:0f 1G 9216 802.3ad
Bond Details
------------------ --------
Bond Mode: 802.3ad
Load Balancing: layer3+4
Minimum Links: 1
LACP Sys Priority:
LACP Rate: 1
LACP Bypass: Active
All VLANs on L2 Port
--------------------
1010
Untagged
--------
1010
cl-netstat counters
-------------------
RX_OK RX_ERR RX_DRP RX_OVR TX_OK TX_ERR TX_DRP TX_OVR
----- ------ ------ ------ ----- ------ ------ ------
481 0 0 0 257 0 0 0
LLDP Details
------------
LocalPort RemotePort(RemoteHost)
--------- ---------------------------
swp3 48:b0:2d:82:05:04(server03)
Routing
-------
Interface bond1 is up, line protocol is up
Link ups: 1 last: 2023/04/17 11:51:10.75
Link downs: 0 last: (never)
PTM status: disabled
vrf: default
index 10 metric 0 mtu 9216 speed 1000
flags: <UP,BROADCAST,RUNNING,MULTICAST>
Type: Ethernet
HWaddr: 48:b0:2d:84:e8:0f
Interface Type bond
Master interface: br_default PVID: 1010
EVPN-MH: ES id 1 ES sysmac 00:00:00:00:00:bb
protodown: off (n/a)
ARP-ND redirect enabled: ARP 1 ND 8
cumulus@leaf04:mgmt:~$ net show bgp evpn vni 5001
VNI: 5001 (known to the kernel)
Type: L3
Tenant VRF: RED
RD: 10.10.20.2:8
Originator IP: 10.10.20.2
Advertise-gw-macip : n/a
Advertise-svi-macip : n/a
Advertise-pip: Yes
System-IP: 10.10.20.2
System-MAC: 44:38:39:22:bb:09
Router-MAC: 44:38:39:22:bb:09
Import Route Target:
0:4001
0:5001
Export Route Target:
65202:5001
cumulus@leaf04:mgmt:~$ net show bgp evpn vni 5002
VNI: 5002 (known to the kernel)
Type: L3
Tenant VRF: GREEN
RD: 10.10.20.2:7
Originator IP: 10.10.20.2
Advertise-gw-macip : n/a
Advertise-svi-macip : n/a
Advertise-pip: Yes
System-IP: 10.10.20.2
System-MAC: 44:38:39:22:bb:09
Router-MAC: 44:38:39:22:bb:09
Import Route Target:
0:4002
0:5002
Export Route Target:
65202:5002
cumulus@leaf04:mgmt:~$ net show evpn mac vni all
VNI 1010 #MACs (local and remote) 5
Flags: B=bypass N=sync-neighs, I=local-inactive, P=peer-active, X=peer-proxy
MAC Type Flags Intf/Remote ES/VTEP VLAN Seq #'s
48:b0:2d:72:28:ff local P bond1 1010 0/0
48:b0:2d:82:05:04 local P bond1 1010 0/0
b6:4b:0f:ea:f2:02 local NP bond1 1010 0/0
44:38:39:22:bb:09 local vlan1010 0/0
44:38:39:22:bb:08 remote 10.10.20.1 0/0
VNI 2020 #MACs (local and remote) 5
Flags: B=bypass N=sync-neighs, I=local-inactive, P=peer-active, X=peer-proxy
MAC Type Flags Intf/Remote ES/VTEP VLAN Seq #'s
ee:54:69:be:3a:3f local NP bond2 2020 0/0
48:b0:2d:75:8e:7a local P bond2 2020 0/0
44:38:39:22:bb:09 local vlan2020 0/0
48:b0:2d:a7:e2:6e local P bond2 2020 0/0
44:38:39:22:bb:08 remote 10.10.20.1 0/0
验证网桥 `br_default` 正在学习 MAC 地址条目
cumulus@leaf03:mgmt:~$ nv show bridge domain br_default mac-table
age bridge-domain entry-type interface last-update MAC address src-vni vlan vni Summary
-- --- ------------- ---------- ---------- ----------- ----------------- ------- ---- ---- ----------------------
0 27 br_default static bond2 725 48:b0:2d:75:8e:7a 2020
1 867 br_default static bond2 867 48:b0:2d:a7:e2:6e 2020
2 63 br_default static bond2 867 ee:54:69:be:3a:3f 2020
3 874 br_default permanent bond2 874 48:b0:2d:d5:d5:91
4 27 br_default static bond1 725 48:b0:2d:72:28:ff 1010
5 867 br_default static bond1 867 48:b0:2d:82:05:04 1010
6 189 br_default static bond1 867 b6:4b:0f:ea:f2:02 1010
7 874 br_default permanent bond1 874 48:b0:2d:7d:9c:74
8 725 br_default vxlan48 189 44:38:39:22:bb:09 1010 None remote-dst: 10.10.20.2
9 874 br_default permanent vxlan48 874 ca:e9:d5:f3:4b:b9 None
10 725 permanent vxlan48 2 00:00:00:00:00:00 2020 None remote-dst: 10.10.20.2
11 permanent br_default 00:00:5e:00:01:14
12 874 br_default permanent br_default 874 44:38:39:22:bb:08 1010
cumulus@leaf04:mgmt:~$ nv show bridge domain br_default mac-table
age bridge-domain entry-type interface last-update MAC address src-vni vlan vni Summary
-- --- ------------- ---------- ---------- ----------- ----------------- ------- ---- ---- ----------------------
0 725 br_default static bond2 725 48:b0:2d:75:8e:7a 2020
1 2 br_default static bond2 725 48:b0:2d:a7:e2:6e 2020
2 180 br_default static bond2 725 ee:54:69:be:3a:3f 2020
3 729 br_default permanent bond2 729 48:b0:2d:95:3f:b0
4 725 br_default vxlan48 725 44:38:39:22:bb:08 2020 None remote-dst: 10.10.20.1
5 729 br_default permanent vxlan48 729 52:d0:7e:cf:6e:ac None
6 725 permanent vxlan48 43 00:00:00:00:00:00 1010 None remote-dst: 10.10.20.1
7 725 br_default static bond1 725 48:b0:2d:72:28:ff 1010
8 59 br_default static bond1 725 b6:4b:0f:ea:f2:02 1010
9 2 br_default static bond1 725 48:b0:2d:82:05:04 1010
10 729 br_default permanent bond1 729 48:b0:2d:84:e8:0f
11 permanent br_default 00:00:5e:00:01:14
12 729 br_default permanent br_default 729 44:38:39:22:bb:09 1010
从上表中，找到三层 VLAN 接口 MAC 地址和 VRR MAC 地址
cumulus@leaf03:mgmt:~$ nv show int vlan1010 | grep mac
mac-address auto
mac-id none
mac 44:38:39:22:bb:08
cumulus@leaf03:mgmt:~$ nv show int vlan1010 ip vrr
operational applied
----------- ----------------- -----------------
enable on
mac-address 00:00:5e:00:01:14 auto
mac-id none
[address] 192.168.10.100/24 192.168.10.100/24
state up up
cumulus@leaf04:mgmt:~$ nv show int vlan1010 | grep mac
mac-address auto
mac-id none
mac 44:38:39:22:bb:09
cumulus@leaf04:mgmt:~$ nv show int vlan1010 ip vrr
operational applied
----------- ----------------- -----------------
enable on
mac-address 00:00:5e:00:01:14 auto
mac-id none
[address] 192.168.10.100/24 192.168.10.100/24
state up up
验证入口 PE (leaf03) 上连接到 leaf01 和 leaf02 的终端主机 *192.168.1.10* 的 EVPN type-5 路由
cumulus@leaf03:mgmt:~$ net show bgp evpn route type 5 | grep 192.168.1\. -A 4 -B 1
Route Distinguisher: 10.10.10.1:6
* [5]:[0]:[24]:[192.168.1.0] RD 10.10.10.1:6
10.10.10.1 (spine03)
0 65299 65210 65110 65199 65101 ?
RT:65101:4001 ET:8 Rmac:44:38:39:22:bb:06
*> [5]:[0]:[24]:[192.168.1.0] RD 10.10.10.1:6
10.10.10.1 (spine04)
0 65299 65210 65110 65199 65101 ?
RT:65101:4001 ET:8 Rmac:44:38:39:22:bb:06
Route Distinguisher: 10.10.10.2:4
* [5]:[0]:[24]:[192.168.1.0] RD 10.10.10.2:4
10.10.10.2 (spine03)
0 65299 65210 65110 65199 65102 ?
RT:65102:4001 ET:8 Rmac:44:38:39:22:bb:07
*> [5]:[0]:[24]:[192.168.1.0] RD 10.10.10.2:4
10.10.10.2 (spine04)
0 65299 65210 65110 65199 65102 ?
RT:65102:4001 ET:8 Rmac:44:38:39:22:bb:07
Route Distinguisher: 10.10.10.2:5
验证出口 PE (leaf01) 上连接到 leaf01 和 leaf02 的终端主机 *192.168.1.10* 的 EVPN type-5 路由
cumulus@leaf01:mgmt:~$ net show bgp evpn route type 5 | grep 192.168.1\. -A 4 -B 1
Route Distinguisher: 10.10.10.1:6
*> [5]:[0]:[24]:[192.168.1.0] RD 10.10.10.1:6
10.10.10.1 (leaf01)
0 32768 ?
ET:8 RT:65101:4001 Rmac:44:38:39:22:bb:06
Route Distinguisher: 10.10.10.2:4
* [5]:[0]:[24]:[192.168.1.0] RD 10.10.10.2:4
10.10.10.2 (spine02)
0 65199 65102 ?
RT:65102:4001 ET:8 Rmac:44:38:39:22:bb:07
*> [5]:[0]:[24]:[192.168.1.0] RD 10.10.10.2:4
10.10.10.2 (spine01)
0 65199 65102 ?
RT:65102:4001 ET:8 Rmac:44:38:39:22:bb:07
Route Distinguisher: 10.10.10.2:5
--
查看来自边界叶节点的路由
cumulus@borderleaf04:mgmt:~$ net show bgp sum
show bgp ipv4 unicast summary
=============================
BGP router identifier 10.10.20.11, local AS number 65210 vrf-id 0
BGP table version 18
RIB entries 19, using 3800 bytes of memory
Peers 3, using 68 KiB of memory
Peer groups 2, using 128 bytes of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
borderleaf01(swp3) 4 65110 1260 1263 0 0 0 01:01:26 5 10
spine03(swp1) 4 65299 1341 1345 0 0 0 01:01:24 4 10
spine04(swp2) 4 65299 1340 1343 0 0 0 01:01:16 4 10
Total number of neighbors 3
show bgp ipv6 unicast summary
=============================
% No BGP neighbors found
show bgp l2vpn evpn summary
===========================
BGP router identifier 10.10.20.11, local AS number 65210 vrf-id 0
BGP table version 0
RIB entries 35, using 7000 bytes of memory
Peers 3, using 68 KiB of memory
Peer groups 2, using 128 bytes of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
borderleaf01(swp3) 4 65110 1260 1263 0 0 0 01:01:26 4 8
spine03(swp1) 4 65299 1342 1346 0 0 0 01:01:24 42 46
spine04(swp2) 4 65299 1340 1343 0 0 0 01:01:16 42 46
Total number of neighbors 3
cumulus@borderleaf04:mgmt:~$ net show bgp vrf RED
show bgp vrf RED ipv4 unicast
=============================
BGP table version is 11, local router ID is 10.10.20.11, vrf id 13
Default local pref 100, local AS 65210
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*= 192.168.1.0/24 10.10.10.2< 0 65110 65199 65102 ?
*> 10.10.10.1< 0 65110 65199 65101 ?
*= 192.168.10.0/24 10.10.20.2< 0 65299 65202 ?
* 10.10.20.2< 0 65299 65202 ?
*> 10.10.20.1< 0 65299 65201 ?
* 10.10.20.1< 0 65299 65201 ?
*= 192.168.10.110/32
10.10.20.2< 0 65299 65202 i
* 10.10.20.2< 0 65299 65202 i
*> 10.10.20.1< 0 65299 65201 i
* 10.10.20.1< 0 65299 65201 i
Displayed 3 routes and 10 total paths
show bgp vrf RED ipv6 unicast
=============================
No BGP prefixes displayed, 0 exist
cumulus@borderleaf04:mgmt:~$ net show bgp evpn route type 5 | grep 192.168.1\. -A 3 -B 1
Route Distinguisher: 10.10.10.1:6
*> [5]:[0]:[24]:[192.168.1.0] RD 10.10.10.1:6
10.10.10.1 (borderleaf01)
0 65110 65199 65101 ?
RT:65101:4001 ET:8 Rmac:44:38:39:22:bb:06
Route Distinguisher: 10.10.10.2:4
*> [5]:[0]:[24]:[192.168.1.0] RD 10.10.10.2:4
10.10.10.2 (borderleaf01)
0 65110 65199 65102 ?
RT:65102:4001 ET:8 Rmac:44:38:39:22:bb:07
--
Route Distinguisher: 10.10.20.1:8
* [5]:[0]:[24]:[192.168.10.0] RD 10.10.20.1:8
10.10.20.1 (spine03)
0 65299 65201 ?
RT:65201:5001 ET:8 Rmac:44:38:39:22:bb:08
*> [5]:[0]:[24]:[192.168.10.0] RD 10.10.20.1:8
10.10.20.1 (spine04)
0 65299 65201 ?
RT:65201:5001 ET:8 Rmac:44:38:39:22:bb:08
--
Route Distinguisher: 10.10.20.2:8
* [5]:[0]:[24]:[192.168.10.0] RD 10.10.20.2:8
10.10.20.2 (spine04)
0 65299 65202 ?
RT:65202:5001 ET:8 Rmac:44:38:39:22:bb:09
*> [5]:[0]:[24]:[192.168.10.0] RD 10.10.20.2:8
10.10.20.2 (spine03)
0 65299 65202 ?
RT:65202:5001 ET:8 Rmac:44:38:39:22:bb:09