Cumulus Linux 以太网存储结构配置指南
NVIDIA Spectrum 交换机与 Cumulus Linux 网络操作系统相结合，使您能够创建可预测、低延迟、高吞吐量的网络。一流的硬件和软件组合使您能够以最小的努力部署高效、冗余且 ESF (以太网存储结构) 优化的网络。ESF 包括完整的、混合的或 HCI 存储结构以及 AI 系统的存储后端。
本配置指南演示了在 NVIDIA Spectrum 交换机上使用 Cumulus Linux 操作系统和 NVIDIA 用户体验 - NVUE 命令行界面 (CLI) 进行 ESF 的通用网络配置。NVIDIA 还提供 NVIDIA DGX POD 参考架构解决方案产品组合，并与所有领先合作伙伴合作。
您可以使用 Cumulus Linux 集来配置具有 多机箱链路聚合 (MLAG) 和 虚拟路由器冗余 (VRR) 的冗余双活二层和三层环境。
具有基于 Fabric 的 NVMe (NVMeOF) 或 GPUDirect 技术的 ESF 环境通常利用基于融合以太网的 RDMA (RoCE) 来加速节点之间密集的数据移动，从而消除延迟密集型数据复制、CPU 或 GPU 中断以及上下文切换。完全共享的缓冲区架构使 NVIDIA Spectrum 交换机能够为 RoCE 流量提供公平、可预测的线速吞吐量和超低端口到端口延迟。大多数 RoCE 配置都包含复杂的步骤，手动配置容易出错。Cumulus Linux 将 RoCE 配置简化为单个命令，并自动应用最佳实践设置以获得最佳性能。
此外，NVIDIA Spectrum ASIC 遥测功能，包括独特的“刚刚发生”(WJH) 功能，可提供对网络的实时可见性。您可以查看流延迟事件、缓冲区占用率和丢包情况，以发现、诊断和解决网络问题。WJH 监控支持所有事件和流量类型，包括第 1 层、第 2 层、ACL、第 3 层、隧道、缓冲区和 RoCE。
网络模拟对于避免手动 CLI 以及复制和粘贴配置错误至关重要。NVIDIA Air 提供了一个基于云的在线网络模拟平台，您可以在其中创建物理环境的数字孪生。您可以在 AIR 中构建、模拟和体验 Cumulus Linux 交换机，同时在将部署配置应用于生产环境之前对其进行测试。查看 NVIDIA Air 用户指南 以了解更多详细信息。
每个存储供应商的部署可能都有不同的规范。您可以随时查看 Cumulus Linux 文档，以探索更多详细信息并匹配您的部署配置。
Cumulus Linux 文档为某些功能（如 MLAG、VRR、VXLAN 和 EVPN）提供了预构建的 Try It 演示。这些演示会自动在 Air 中启动网络模拟，以便您可以练习配置。
如果您的交换机仍在运行 NVIDIA Onyx，请按照本指南迁移到 Cumulus Linux，以 卸载 Onyx 映像 并 安装新的 Cumulus Linux 映像。
为了简化从 ONYX 到 NVUE 的过渡，您可以使用此 NVUE 迁移工具 上传 Onyx 交换机中的运行配置文件。
部署类型
存储供应商在部署 ESF 时，通常在存储阵列和机架顶部（ToR 或叶）交换机之间使用二层连接。
以下是三种主要的网络配置
- MLAG - 二层 ToR，具有路由（三层）上行链路到 Spine 交换机。
- MLAG over MLAG（也称为背靠背 MLAG）- 二层 ToR，具有交换（二层）上行链路到 Spine 交换机。 此场景可用于较大的二层域，但最多只能有两个 Spine 交换机。 您可以在两个级别配置 MLAG，即 ToR 和 Spine 交换机上。 路由在 Spine 交换机上。
- 双活 VXLAN - 使用 VXLAN 和 EVPN 在三层网络上扩展的二层域。 此类部署可以使用 VXLAN 二层扩展或路由配置。
MLAG 是一种双活二层冗余协议，可在使带宽加倍的同时实现冗余 ToR 二层网络。但是，它还使您能够利用 VRR，即双活路由实例。借助 VRR，ToR（或 Spine）交换机都可以同时路由存储流量，并创建冗余、更高带宽的三层网络。
无论您选择哪种 ESF 网络设计，当涉及到 ToR 二层冗余时，一些供应商更喜欢在存储网络接口卡 (NIC) 上部署双活绑定 (LACP/xor)，而另一些供应商更喜欢部署主备（或 alb）。以下示例对二层或三层冗余（或两者）都使用 MLAG 协议。
对于 RoCEv1，您必须在所有地方设置二层 Fabric，因此不需要路由 (VRR)。对于 RoCEv2，如果您的网络需要子网间路由，您可以部署具有 VRR 网关（以及带 EVPN 的 BGP）的三层 Fabric。
MLAG
本节介绍如何使用 NVUE 在 Cumulus Linux 上配置 MLAG。
要使用 MLAG 实现冗余的机架顶部部署，您必须使用双活绑定配置存储阵列 NIC。即使对于朝向 ToR 交换机的非双活绑定，以下示例也使用 MLAG 协议集
- 将 VRR 用作存储流量默认网关以路由到其他子网。
- 在 MLAG over MLAG（背靠背 MLAG）部署中，ToR 上行链路设置为 MLAG 端口。
在配置 MLAG 之前，请确保两个对等方都运行相同的 Cumulus Linux 版本。NVIDIA 建议您运行最新的 Cumulus Linux 版本。要验证交换机上的 Cumulus Linux 版本，请运行 cat /etc/lsb-release 或 nv show system 命令。
机箱间 Bond（对等链路）
MLAG 需要对等方之间的专用链路（bond）。Cumulus Linux 将保留名称 peerlink 用于此 bond，并使用唯一的 VLAN 三层子接口 peerlink.4094。对等链路维护 MLAG 对等方之间的状态信息，并在发生故障时充当流量备份路径。
在规划链路故障时，您需要设置具有足够带宽的对等链路，以满足您的站点处理故障场景的策略。NVIDIA 建议您确定通过单连接接口发送的带宽量，并将该带宽的一半设置为对等链路。
默认情况下，Cumulus Linux 允许对等链路上的所有 VLAN，并在您创建对等链路时自动添加它们。当您运行以下命令时，Cumulus Linux 还会自动创建 peerlink.4094 子接口。接口 IPv6 链路本地地址在对等方之间提供三层连接。
要创建对等链路和 peerlink.4094 接口
cumulus@switch:mgmt:~$ nv set interface peerlink bond member <interface-list>
cumulus@switch:mgmt:~$ nv set mlag peer-ip linklocal
请勿使用 169.254.0.1 作为 MLAG 对等 IP 地址。Cumulus Linux 将此地址用于 BGP unnumbered 接口。
MLAG 备份 IP 地址
当对等链路断开时，MLAG 使用备份 IP 地址来维护 MLAG 域。
要设置 MLAG 备份 IP 地址
cumulus@switch:mgmt:~$ nv set mlag backup [peer-address] vrf mgmt
确保 MLAG 备份 IP 地址与 peerlink.4094 地址不同，并且可以从网络访问。使用交换机环回或管理 IP 地址。
MLAG 系统 MAC 地址
MLAG 系统 MAC 地址在各种控制协议（例如生成树和 LACP）中将两个对等方表示为同一交换机。确保在两个对等方上设置相同的 MLAG 系统 MAC 地址。
要设置 MLAG 系统 MAC 地址
cumulus@switch:mgmt:~$ nv set mlag mac-address <address>
仅使用保留范围 44:38:39:ff:00:00-44:38:39:ff:ff:ff 中的 MLAG MAC 地址。始终为不同的 MLAG 对设置唯一的 MAC 地址。
如果您的 ESF 环境基于双活 VXLAN 和 EVPN Fabric，您还必须在两个对等方上设置系统全局任播 MAC 地址
cumulus@switch:mgmt:~$ nv set system global anycast-mac <address>
MLAG 端口
MLAG 端口是在两个 MLAG 对等方上设置的 bond 接口，其中包含连接到存储网卡的接口作为其成员。您必须为每个 MLAG 端口指定唯一的 MLAG ID（在两个对等方上）。默认情况下，Cumulus Linux 使用 LACP (802.3ad) bond 模式，但 balance-xor 模式也是一种选择。
MLAG 端口配置取决于存储阵列 NIC 上的绑定类型。例如，当 NIC 设置为双活绑定模式 (LACP/xor) 时，您必须在连接到它们的 ToR 交换机上配置 MLAG 端口。如果阵列 NIC 设置为主备（或 alb）绑定，则不要设置 MLAG 端口；流量仅通过单个交换机，直到发生链路故障或不同的流或应用程序传输。
要配置 MLAG 端口，请创建具有成员的 bond 接口并设置 MLAG ID。如果您不需要 MLAG 端口，请跳过此步骤。
cumulus@switch:mgmt:~$ nv set interface <bond-name> bond member <interface-list>
cumulus@switch:mgmt:~$ nv set interface <bond-name> bond mlag id <number>
由于大多数 ESF 环境都使用巨型 MTU 设置，因此 Cumulus Linux 默认 MTU 对于所有物理和逻辑接口均为 9216B。要更改 MTU
cumulus@switch:mgmt:~$ nv set interface <interface-list> link mtu <mtu>
您可以通过设置 MLAG 优先级来静态定义 MLAG 主设备和从设备
cumulus@switch:mgmt:~$ nv set mlag priority <priority>
在重新启动后推迟 MLAG 运行（以便服务器在交换机之前启动）
cumulus@switch:mgmt:~$ nv set mlag init-delay <seconds>
有关 MLAG 的更多信息，请参阅 此文档。
MLAG over MLAG（背靠背 MLAG）
有时 ESF 环境需要为许多主机提供二层连接，在这种情况下，您需要的 ToR 交换机超过两个。在这种情况下，ESF 环境扩展为叶-Spine 拓扑以扩展二层域。有两种类型的网络部署：MLAG over MLAG 或在具有 VXLAN 和 EVPN 的三层 Fabric 上扩展的二层域。
对于 MLAG over MLAG，您还需要在 Spine 交换机上设置 MLAG，并在 ToR 和 Spine 交换机之间配置 MLAG 端口。
Spine 交换机上的 MLAG 配置与 ToR 交换机上的配置相同。上图显示，ToR 和 Spine 交换机之间的所有物理端口都绑定到一个 MLAG 端口中。此 bond 必须在每个 MLAG 对等方上具有相同的 MLAG ID（ToR 和 Spine 交换机上的 ID 可以不同）。
网桥和 VLAN
以太网网桥通过将物理和逻辑接口连接到单个二层域，从而在 Cumulus Linux 交换机上启用二层。但是，Cumulus Linux 默认情况下没有网桥，所有物理端口都已路由。因此，您必须创建一个 VLAN 感知网桥，以在存储网卡和 ToR 交换机之间启用二层通信。该网桥实现所有二层技术，如 STP、MAC 地址学习、VLAN、中继等。
创建网桥后，您需要分配端口（swp/MLAG 端口）。Cumulus Linux 会自动将端口添加到 VLAN 1 中作为非标记 VLAN。
要创建网桥并将端口分配给网桥
cumulus@switch:mgmt:~$ nv set bridge domain br_default
cumulus@switch:mgmt:~$ nv set interface <interface> bridge domain br_default
在大多数情况下，您不需要在交换机上使用默认 VLAN1。要向网桥添加新 VLAN 并将端口设置为访问（非标记）端口
cumulus@switch:mgmt:~$ nv set bridge domain br_default vlan <vlan-list>
cumulus@switch:mgmt:~$ nv set interface <interface> bridge domain br_default access <vlan-id>
如果存储阵列 NIC 使用 dot1q，请将 ToR 下行链路设置为中继端口。每个中继端口都使用 dot1q 标记来处理指定的 VLAN 和本机 VLAN（非标记）。
要将端口设置为具有本机 VLAN 的中继端口
cumulus@switch:mgmt:~$ nv set interface <interface> bridge domain br_default vlan <vlan-list>
cumulus@switch:mgmt:~$ nv set interface <interface> bridge domain br_default untagged <vlan-id>
在部署 MLAG over MLAG 时，网桥和 VLAN 的配置对于 MLAG 的两个层（域）都是相同的。如果您在存储网卡上使用多个 VLAN，请在 ToR 和 Spine 交换机之间的 MLAG 端口上使用相同的中继配置。
MLAG 配置示例
cumulus@leaf01:mgmt:~$ nv set interface peerlink bond member swp2-3
cumulus@leaf01:mgmt:~$ nv set mlag peer-ip linklocal
cumulus@leaf01:mgmt:~$ nv set mlag backup 192.168.200.12 vrf mgmt
cumulus@leaf01:mgmt:~$ nv set mlag mac-address 44:38:39:BE:EF:AA
cumulus@leaf01:mgmt:~$ nv set mlag priority 1000
cumulus@leaf01:mgmt:~$ nv set mlag init-delay 10
cumulus@leaf01:mgmt:~$ nv set interface bond1 bond member swp1
cumulus@leaf01:mgmt:~$ nv set interface bond1 bond mlag id 1
cumulus@leaf01:mgmt:~$ nv set interface bond1 link mtu 9000
cumulus@leaf01:mgmt:~$ nv set bridge domain br_default
cumulus@leaf01:mgmt:~$ nv set bridge domain br_default vlan 10,20,30
cumulus@leaf01:mgmt:~$ nv set interface bond1 bridge domain br_default
cumulus@leaf01:mgmt:~$ nv set interface bond1 bridge domain br_default vlan 10,20,30
cumulus@leaf01:mgmt:~$ nv config apply
cumulus@leaf01:mgmt:~$ nv config save
cumulus@leaf02:mgmt:~$ nv set interface peerlink bond member swp2-3
cumulus@leaf02:mgmt:~$ nv set mlag peer-ip linklocal
cumulus@leaf02:mgmt:~$ nv set mlag backup 192.168.200.11 vrf mgmt
cumulus@leaf02:mgmt:~$ nv set mlag mac-address 44:38:39:BE:EF:AA
cumulus@leaf02:mgmt:~$ nv set mlag init-delay 10
cumulus@leaf02:mgmt:~$ nv set interface bond1 bond member swp1
cumulus@leaf02:mgmt:~$ nv set interface bond1 bond mlag id 1
cumulus@leaf02:mgmt:~$ nv set interface bond1 link mtu 9000
cumulus@leaf02:mgmt:~$ nv set bridge domain br_default
cumulus@leaf02:mgmt:~$ nv set bridge domain br_default vlan 10,20,30
cumulus@leaf02:mgmt:~$ nv set interface bond1 bridge domain br_default
cumulus@leaf02:mgmt:~$ nv set interface bond1 bridge domain br_default vlan 10,20,30
cumulus@leaf02:mgmt:~$ nv config apply
cumulus@leaf02:mgmt:~$ nv config save
cumulus@leaf01:mgmt:~$ nv set interface peerlink bond member swp2-3
cumulus@leaf01:mgmt:~$ nv set mlag peer-ip linklocal
cumulus@leaf01:mgmt:~$ nv set mlag backup 192.168.200.12 vrf mgmt
cumulus@leaf01:mgmt:~$ nv set mlag mac-address 44:38:39:BE:EF:AA
cumulus@leaf01:mgmt:~$ nv set mlag priority 1000
cumulus@leaf01:mgmt:~$ nv set mlag init-delay 10
cumulus@leaf01:mgmt:~$ nv set interface swp1 link mtu 9000
cumulus@leaf01:mgmt:~$ nv set bridge domain br_default
cumulus@leaf01:mgmt:~$ nv set bridge domain br_default vlan 10,20,30
cumulus@leaf01:mgmt:~$ nv set interface swp1 bridge domain br_default
cumulus@leaf01:mgmt:~$ nv set interface swp1 bridge domain br_default vlan 10,20,30
cumulus@leaf01:mgmt:~$ nv config apply
cumulus@leaf01:mgmt:~$ nv config save
cumulus@leaf02:mgmt:~$ nv set interface peerlink bond member swp2-3
cumulus@leaf02:mgmt:~$ nv set mlag peer-ip linklocal
cumulus@leaf02:mgmt:~$ nv set mlag backup 192.168.200.11 vrf mgmt
cumulus@leaf02:mgmt:~$ nv set mlag mac-address 44:38:39:BE:EF:AA
cumulus@leaf02:mgmt:~$ nv set mlag init-delay 10
cumulus@leaf02:mgmt:~$ nv set interface swp1 link mtu 9000
cumulus@leaf02:mgmt:~$ nv set bridge domain br_default
cumulus@leaf02:mgmt:~$ nv set bridge domain br_default vlan 10,20,30
cumulus@leaf02:mgmt:~$ nv set interface swp1 bridge domain br_default
cumulus@leaf02:mgmt:~$ nv set interface swp1 bridge domain br_default vlan 10,20,30
cumulus@leaf02:mgmt:~$ nv config apply
cumulus@leaf02:mgmt:~$ nv config save
cumulus@leaf01:mgmt:~$ nv set interface peerlink bond member swp2-3
cumulus@leaf01:mgmt:~$ nv set mlag peer-ip linklocal
cumulus@leaf01:mgmt:~$ nv set mlag backup 192.168.200.12 vrf mgmt
cumulus@leaf01:mgmt:~$ nv set mlag mac-address 44:38:39:BE:EF:AA
cumulus@leaf01:mgmt:~$ nv set mlag priority 1000
cumulus@leaf01:mgmt:~$ nv set mlag init-delay 10
cumulus@leaf01:mgmt:~$ nv set interface bond1 bond member swp1
cumulus@leaf01:mgmt:~$ nv set interface bond2 bond member swp4-swp5
cumulus@leaf01:mgmt:~$ nv set interface bond1 bond mlag id 1
cumulus@leaf01:mgmt:~$ nv set interface bond2 bond mlag id 2
cumulus@leaf01:mgmt:~$ nv set interface bond1,bond2 link mtu 9000
cumulus@leaf01:mgmt:~$ nv set bridge domain br_default
cumulus@leaf01:mgmt:~$ nv set bridge domain br_default vlan 10,20,30
cumulus@leaf01:mgmt:~$ nv set interface bond1,bond2 bridge domain br_default
cumulus@leaf01:mgmt:~$ nv set interface bond1,bond2 bridge domain br_default vlan 10,20,30
cumulus@leaf01:mgmt:~$ nv config apply
cumulus@leaf01:mgmt:~$ nv config save
cumulus@leaf02:mgmt:~$ nv set interface peerlink bond member swp2-3
cumulus@leaf02:mgmt:~$ nv set mlag peer-ip linklocal
cumulus@leaf02:mgmt:~$ nv set mlag backup 192.168.200.11 vrf mgmt
cumulus@leaf02:mgmt:~$ nv set mlag mac-address 44:38:39:BE:EF:AA
cumulus@leaf02:mgmt:~$ nv set mlag init-delay 10
cumulus@leaf02:mgmt:~$ nv set interface bond1 bond member swp1
cumulus@leaf02:mgmt:~$ nv set interface bond2 bond member swp4-swp5
cumulus@leaf02:mgmt:~$ nv set interface bond1 bond mlag id 1
cumulus@leaf02:mgmt:~$ nv set interface bond2 bond mlag id 2
cumulus@leaf02:mgmt:~$ nv set interface bond1,bond2 link mtu 9000
cumulus@leaf02:mgmt:~$ nv set bridge domain br_default
cumulus@leaf02:mgmt:~$ nv set bridge domain br_default vlan 10,20,30
cumulus@leaf02:mgmt:~$ nv set interface bond1,bond2 bridge domain br_default
cumulus@leaf02:mgmt:~$ nv set interface bond1,bond2 bridge domain br_default vlan 10,20,30
cumulus@leaf02:mgmt:~$ nv config apply
cumulus@leaf02:mgmt:~$ nv config save
cumulus@spine01:mgmt:~$ nv set interface peerlink bond member swp3-4
cumulus@spine01:mgmt:~$ nv set mlag peer-ip linklocal
cumulus@spine01:mgmt:~$ nv set mlag backup 192.168.200.14 vrf mgmt
cumulus@spine01:mgmt:~$ nv set mlag mac-address 44:38:39:BE:EF:BB
cumulus@spine01:mgmt:~$ nv set mlag priority 1000
cumulus@spine01:mgmt:~$ nv set mlag init-delay 10
cumulus@spine01:mgmt:~$ nv set interface bond3 bond member swp1-2
cumulus@spine01:mgmt:~$ nv set interface bond3 bond mlag id 3
cumulus@spine01:mgmt:~$ nv set interface bond3 link mtu 9000
cumulus@spine01:mgmt:~$ nv set bridge domain br_default
cumulus@spine01:mgmt:~$ nv set bridge domain br_default vlan 10,20,30
cumulus@spine01:mgmt:~$ nv set interface bond3 bridge domain br_default
cumulus@spine01:mgmt:~$ nv set interface bond3 bridge domain br_default vlan 10,20,30
cumulus@spine01:mgmt:~$ nv config apply
cumulus@spine01:mgmt:~$ nv config save
cumulus@spine02:mgmt:~$ nv set interface peerlink bond member swp3-4
cumulus@spine02:mgmt:~$ nv set mlag peer-ip linklocal
cumulus@spine02:mgmt:~$ nv set mlag backup 192.168.200.13 vrf mgmt
cumulus@spine02:mgmt:~$ nv set mlag mac-address 44:38:39:BE:EF:BB
cumulus@spine02:mgmt:~$ nv set mlag init-delay 10
cumulus@spine02:mgmt:~$ nv set interface bond3 bond member swp1-2
cumulus@spine02:mgmt:~$ nv set interface bond3 bond mlag id 3
cumulus@spine02:mgmt:~$ nv set interface bond3 link mtu 9000
cumulus@spine02:mgmt:~$ nv set bridge domain br_default
cumulus@spine02:mgmt:~$ nv set bridge domain br_default vlan 10,20,30
cumulus@spine02:mgmt:~$ nv set interface bond3 bridge domain br_default
cumulus@spine02:mgmt:~$ nv set interface bond3 bridge domain br_default vlan 10,20,30
cumulus@spine02:mgmt:~$ nv config apply
cumulus@spine02:mgmt:~$ nv config save
cumulus@leaf01:mgmt:~$ nv set interface peerlink bond member swp2-3
cumulus@leaf01:mgmt:~$ nv set mlag peer-ip linklocal
cumulus@leaf01:mgmt:~$ nv set mlag backup 192.168.200.12 vrf mgmt
cumulus@leaf01:mgmt:~$ nv set mlag mac-address 44:38:39:BE:EF:AA
cumulus@leaf01:mgmt:~$ nv set mlag priority 1000
cumulus@leaf01:mgmt:~$ nv set mlag init-delay 10
cumulus@leaf01:mgmt:~$ nv set interface bond2 bond member swp4-swp5
cumulus@leaf01:mgmt:~$ nv set interface bond2 bond mlag id 2
cumulus@leaf01:mgmt:~$ nv set interface swp1,bond2 link mtu 9000
cumulus@leaf01:mgmt:~$ nv set bridge domain br_default
cumulus@leaf01:mgmt:~$ nv set bridge domain br_default vlan 10,20,30
cumulus@leaf01:mgmt:~$ nv set interface swp1,bond2 bridge domain br_default
cumulus@leaf01:mgmt:~$ nv set interface swp1,bond2 bridge domain br_default vlan 10,20,30
cumulus@leaf01:mgmt:~$ nv config apply
cumulus@leaf01:mgmt:~$ nv config save
cumulus@leaf02:mgmt:~$ nv set interface peerlink bond member swp2-3
cumulus@leaf02:mgmt:~$ nv set mlag peer-ip linklocal
cumulus@leaf02:mgmt:~$ nv set mlag backup 192.168.200.11 vrf mgmt
cumulus@leaf02:mgmt:~$ nv set mlag mac-address 44:38:39:BE:EF:AA
cumulus@leaf02:mgmt:~$ nv set mlag init-delay 10
cumulus@leaf02:mgmt:~$ nv set interface bond2 bond member swp4-swp5
cumulus@leaf02:mgmt:~$ nv set interface bond2 bond mlag id 2
cumulus@leaf02:mgmt:~$ nv set interface swp1,bond2 link mtu 9000
cumulus@leaf02:mgmt:~$ nv set bridge domain br_default
cumulus@leaf02:mgmt:~$ nv set bridge domain br_default vlan 10,20,30
cumulus@leaf02:mgmt:~$ nv set interface swp1,bond2 bridge domain br_default
cumulus@leaf02:mgmt:~$ nv set interface swp1,bond2 bridge domain br_default vlan 10,20,30
cumulus@leaf02:mgmt:~$ nv config apply
cumulus@leaf02:mgmt:~$ nv config save
cumulus@spine01:mgmt:~$ nv set interface peerlink bond member swp3-4
cumulus@spine01:mgmt:~$ nv set mlag peer-ip linklocal
cumulus@spine01:mgmt:~$ nv set mlag backup 192.168.200.14 vrf mgmt
cumulus@spine01:mgmt:~$ nv set mlag mac-address 44:38:39:BE:EF:BB
cumulus@spine01:mgmt:~$ nv set mlag priority 1000
cumulus@spine01:mgmt:~$ nv set mlag init-delay 10
cumulus@spine01:mgmt:~$ nv set interface bond3 bond member swp1-2
cumulus@spine01:mgmt:~$ nv set interface bond3 bond mlag id 3
cumulus@spine01:mgmt:~$ nv set interface bond3 link mtu 9000
cumulus@spine01:mgmt:~$ nv set bridge domain br_default
cumulus@spine01:mgmt:~$ nv set bridge domain br_default vlan 10,20,30
cumulus@spine01:mgmt:~$ nv set interface bond3 bridge domain br_default
cumulus@spine01:mgmt:~$ nv set interface bond3 bridge domain br_default vlan 10,20,30
cumulus@spine01:mgmt:~$ nv config apply
cumulus@spine01:mgmt:~$ nv config save
cumulus@spine02:mgmt:~$ nv set interface peerlink bond member swp3-4
cumulus@spine02:mgmt:~$ nv set mlag peer-ip linklocal
cumulus@spine02:mgmt:~$ nv set mlag backup 192.168.200.13 vrf mgmt
cumulus@spine02:mgmt:~$ nv set mlag mac-address 44:38:39:BE:EF:BB
cumulus@spine02:mgmt:~$ nv set mlag init-delay 10
cumulus@spine02:mgmt:~$ nv set interface bond3 bond member swp1-2
cumulus@spine02:mgmt:~$ nv set interface bond3 bond mlag id 3
cumulus@spine02:mgmt:~$ nv set interface bond3 link mtu 9000
cumulus@spine02:mgmt:~$ nv set bridge domain br_default
cumulus@spine02:mgmt:~$ nv set bridge domain br_default vlan 10,20,30
cumulus@spine02:mgmt:~$ nv set interface bond3 bridge domain br_default
cumulus@spine02:mgmt:~$ nv set interface bond3 bridge domain br_default vlan 10,20,30
cumulus@spine02:mgmt:~$ nv config apply
cumulus@spine02:mgmt:~$ nv config save
路由配置
VRR 用作存储流量的网关，用于在 ToR 交换机上的 VLAN 之间路由或在数据中心和外部网络中的机架之间路由流量。
根据您的网络部署类型，您可以在 ToR 交换机或 Spine 交换机（MLAG over MLAG）上配置 VRR 实例。对于每个存储子网 (VLAN)，您需要创建一个 VRR 实例。此虚拟路由器实例在交换机虚拟接口 (SVI) 之上工作，并拥有一个虚拟 IP 地址，该地址充当子网的网关，以及虚拟 MAC 地址，用于回复来自 NIC 的 ARP 请求。
VRR 是一种双活路由实例，因此两个 MLAG 对等方都必须具有相同的 VRR 实例配置，才能同时回复 ARP 请求和转发路由流量。VRR 与虚拟路由冗余协议 (VRRP) 不同，因为两个对等方上的实例彼此独立工作，而不是以主备模式工作。
要设置 VRR，首先在每个对等方上创建一个 SVI（VLAN 接口），并在同一子网（作为 NIC 子网）中设置唯一的 IP 地址。然后，在每个 SVI 之上配置 VRR 实例。两个对等方共享相同的虚拟 IP 地址和 MAC 地址。
使用 vlan 关键字后跟 VLAN ID 创建三层 VLAN 接口，并设置唯一的 IP 地址
cumulus@switch:mgmt:~$ nv set interface vlan<ID> ip address <address>
在 SVI 之上添加 VRR 地址并启用虚拟实例
cumulus@switch:mgmt:~$ nv set interface vlan<ID> ip vrr address <address>
cumulus@switch:mgmt:~$ nv set interface vlan<ID> ip vrr mac-address <address>
cumulus@switch:mgmt:~$ nv set interface vlan<ID> ip vrr state up
在为二层域（ToR 和 Spine 交换机）MLAG 对等方设置 VRR 后，您需要配置三层上行链路，即路由器端口或 SVI。然后，您可以使用静态或动态路由来路由流量。数据中心中最常见的路由协议是边界网关协议 (BGP)，尤其是 eBGP。
Cumulus Linux 提供了一种独特且简单的方式来使用 Auto BGP 和 BGP Unnumbered 配置 BGP。
VRR 配置示例
cumulus@leaf01:mgmt:~$ nv set interface peerlink bond member swp2-3
cumulus@leaf01:mgmt:~$ nv set mlag peer-ip linklocal
cumulus@leaf01:mgmt:~$ nv set mlag backup 192.168.200.12 vrf mgmt
cumulus@leaf01:mgmt:~$ nv set mlag mac-address 44:38:39:BE:EF:AA
cumulus@leaf01:mgmt:~$ nv set mlag priority 1000
cumulus@leaf01:mgmt:~$ nv set mlag init-delay 10
cumulus@leaf01:mgmt:~$ nv set interface bond1 bond member swp1
cumulus@leaf01:mgmt:~$ nv set interface bond1 bond mlag id 1
cumulus@leaf01:mgmt:~$ nv set interface bond1 link mtu 9000
cumulus@leaf01:mgmt:~$ nv set bridge domain br_default
cumulus@leaf01:mgmt:~$ nv set bridge domain br_default vlan 10,20,30
cumulus@leaf01:mgmt:~$ nv set interface bond1 bridge domain br_default
cumulus@leaf01:mgmt:~$ nv set interface bond1 bridge domain br_default vlan 10,20,30
cumulus@leaf01:mgmt:~$ nv set interface vlan10 ip address 10.1.10.2/24
cumulus@leaf01:mgmt:~$ nv set interface vlan10 ip ip vrr address 10.1.10.1/24
cumulus@leaf01:mgmt:~$ nv set interface vlan10 ip vrr mac-address 00:00:00:00:00:10
cumulus@leaf01:mgmt:~$ nv set interface vlan10 ip vrr state up
cumulus@leaf01:mgmt:~$ nv set interface vlan20 ip address 10.1.20.2/24
cumulus@leaf01:mgmt:~$ nv set interface vlan20 ip ip vrr address 10.1.20.1/24
cumulus@leaf01:mgmt:~$ nv set interface vlan20 ip vrr mac-address 00:00:00:00:00:20
cumulus@leaf01:mgmt:~$ nv set interface vlan20 ip vrr state up
cumulus@leaf01:mgmt:~$ nv set interface vlan30 ip address 10.1.30.2/24
cumulus@leaf01:mgmt:~$ nv set interface vlan30 ip ip vrr address 10.1.30.1/24
cumulus@leaf01:mgmt:~$ nv set interface vlan30 ip vrr mac-address 00:00:00:00:00:30
cumulus@leaf01:mgmt:~$ nv set interface vlan30 ip vrr state up
cumulus@leaf01:mgmt:~$ nv config apply
cumulus@leaf01:mgmt:~$ nv config save
cumulus@leaf02:mgmt:~$ nv set interface peerlink bond member swp2-3
cumulus@leaf02:mgmt:~$ nv set mlag peer-ip linklocal
cumulus@leaf02:mgmt:~$ nv set mlag backup 192.168.200.11 vrf mgmt
cumulus@leaf02:mgmt:~$ nv set mlag mac-address 44:38:39:BE:EF:AA
cumulus@leaf02:mgmt:~$ nv set mlag init-delay 10
cumulus@leaf02:mgmt:~$ nv set interface bond1 bond member swp1
cumulus@leaf02:mgmt:~$ nv set interface bond1 bond mlag id 1
cumulus@leaf02:mgmt:~$ nv set interface bond1 link mtu 9000
cumulus@leaf02:mgmt:~$ nv set bridge domain br_default
cumulus@leaf02:mgmt:~$ nv set bridge domain br_default vlan 10,20,30
cumulus@leaf02:mgmt:~$ nv set interface bond1 bridge domain br_default
cumulus@leaf02:mgmt:~$ nv set interface bond1 bridge domain br_default vlan 10,20,30
cumulus@leaf02:mgmt:~$ nv set interface vlan10 ip address 10.1.10.3/24
cumulus@leaf02:mgmt:~$ nv set interface vlan10 ip ip vrr address 10.1.10.1/24
cumulus@leaf02:mgmt:~$ nv set interface vlan10 ip vrr mac-address 00:00:00:00:00:10
cumulus@leaf02:mgmt:~$ nv set interface vlan10 ip vrr state up
cumulus@leaf02:mgmt:~$ nv set interface vlan20 ip address 10.1.20.3/24
cumulus@leaf02:mgmt:~$ nv set interface vlan20 ip ip vrr address 10.1.20.1/24
cumulus@leaf02:mgmt:~$ nv set interface vlan20 ip vrr mac-address 00:00:00:00:00:20
cumulus@leaf02:mgmt:~$ nv set interface vlan20 ip vrr state up
cumulus@leaf02:mgmt:~$ nv set interface vlan30 ip address 10.1.30.3/24
cumulus@leaf02:mgmt:~$ nv set interface vlan30 ip ip vrr address 10.1.30.1/24
cumulus@leaf02:mgmt:~$ nv set interface vlan30 ip vrr mac-address 00:00:00:00:00:30
cumulus@leaf02:mgmt:~$ nv set interface vlan30 ip vrr state up
cumulus@leaf02:mgmt:~$ nv config apply
cumulus@leaf02:mgmt:~$ nv config save
cumulus@leaf01:mgmt:~$ nv set interface peerlink bond member swp2-3
cumulus@leaf01:mgmt:~$ nv set mlag peer-ip linklocal
cumulus@leaf01:mgmt:~$ nv set mlag backup 192.168.200.12 vrf mgmt
cumulus@leaf01:mgmt:~$ nv set mlag mac-address 44:38:39:BE:EF:AA
cumulus@leaf01:mgmt:~$ nv set mlag priority 1000
cumulus@leaf01:mgmt:~$ nv set mlag init-delay 10
cumulus@leaf01:mgmt:~$ nv set interface swp1 link mtu 9000
cumulus@leaf01:mgmt:~$ nv set bridge domain br_default
cumulus@leaf01:mgmt:~$ nv set bridge domain br_default vlan 10,20,30
cumulus@leaf01:mgmt:~$ nv set interface swp1 bridge domain br_default
cumulus@leaf01:mgmt:~$ nv set interface swp1 bridge domain br_default vlan 10,20,30
cumulus@leaf01:mgmt:~$ nv set interface vlan10 ip address 10.1.10.2/24
cumulus@leaf01:mgmt:~$ nv set interface vlan10 ip ip vrr address 10.1.10.1/24
cumulus@leaf01:mgmt:~$ nv set interface vlan10 ip vrr mac-address 00:00:00:00:00:10
cumulus@leaf01:mgmt:~$ nv set interface vlan10 ip vrr state up
cumulus@leaf01:mgmt:~$ nv set interface vlan20 ip address 10.1.20.2/24
cumulus@leaf01:mgmt:~$ nv set interface vlan20 ip ip vrr address 10.1.20.1/24
cumulus@leaf01:mgmt:~$ nv set interface vlan20 ip vrr mac-address 00:00:00:00:00:20
cumulus@leaf01:mgmt:~$ nv set interface vlan20 ip vrr state up
cumulus@leaf01:mgmt:~$ nv set interface vlan30 ip address 10.1.30.2/24
cumulus@leaf01:mgmt:~$ nv set interface vlan30 ip ip vrr address 10.1.30.1/24
cumulus@leaf01:mgmt:~$ nv set interface vlan30 ip vrr mac-address 00:00:00:00:00:30
cumulus@leaf01:mgmt:~$ nv set interface vlan30 ip vrr state up
cumulus@leaf01:mgmt:~$ nv config apply
cumulus@leaf01:mgmt:~$ nv config save
cumulus@leaf02:mgmt:~$ nv set interface peerlink bond member swp2-3
cumulus@leaf02:mgmt:~$ nv set mlag peer-ip linklocal
cumulus@leaf02:mgmt:~$ nv set mlag backup 192.168.200.11 vrf mgmt
cumulus@leaf02:mgmt:~$ nv set mlag mac-address 44:38:39:BE:EF:AA
cumulus@leaf02:mgmt:~$ nv set mlag init-delay 10
cumulus@leaf02:mgmt:~$ nv set interface swp1 link mtu 9000
cumulus@leaf02:mgmt:~$ nv set bridge domain br_default
cumulus@leaf02:mgmt:~$ nv set bridge domain br_default vlan 10,20,30
cumulus@leaf02:mgmt:~$ nv set interface swp1 bridge domain br_default
cumulus@leaf02:mgmt:~$ nv set interface swp1 bridge domain br_default vlan 10,20,30
cumulus@leaf02:mgmt:~$ nv set interface vlan10 ip address 10.1.10.3/24
cumulus@leaf02:mgmt:~$ nv set interface vlan10 ip ip vrr address 10.1.10.1/24
cumulus@leaf02:mgmt:~$ nv set interface vlan10 ip vrr mac-address 00:00:00:00:00:10
cumulus@leaf02:mgmt:~$ nv set interface vlan10 ip vrr state up
cumulus@leaf02:mgmt:~$ nv set interface vlan20 ip address 10.1.20.3/24
cumulus@leaf02:mgmt:~$ nv set interface vlan20 ip ip vrr address 10.1.20.1/24
cumulus@leaf02:mgmt:~$ nv set interface vlan20 ip vrr mac-address 00:00:00:00:00:20
cumulus@leaf02:mgmt:~$ nv set interface vlan20 ip vrr state up
cumulus@leaf02:mgmt:~$ nv set interface vlan30 ip address 10.1.30.3/24
cumulus@leaf02:mgmt:~$ nv set interface vlan30 ip ip vrr address 10.1.30.1/24
cumulus@leaf02:mgmt:~$ nv set interface vlan30 ip vrr mac-address 00:00:00:00:00:30
cumulus@leaf02:mgmt:~$ nv set interface vlan30 ip vrr state up
cumulus@leaf02:mgmt:~$ nv config apply
cumulus@leaf02:mgmt:~$ nv config save
cumulus@leaf01:mgmt:~$ nv set interface peerlink bond member swp2-3
cumulus@leaf01:mgmt:~$ nv set mlag peer-ip linklocal
cumulus@leaf01:mgmt:~$ nv set mlag backup 192.168.200.12 vrf mgmt
cumulus@leaf01:mgmt:~$ nv set mlag mac-address 44:38:39:BE:EF:AA
cumulus@leaf01:mgmt:~$ nv set mlag priority 1000
cumulus@leaf01:mgmt:~$ nv set mlag init-delay 10
cumulus@leaf01:mgmt:~$ nv set interface bond1 bond member swp1
cumulus@leaf01:mgmt:~$ nv set interface bond2 bond member swp4-swp5
cumulus@leaf01:mgmt:~$ nv set interface bond1 bond mlag id 1
cumulus@leaf01:mgmt:~$ nv set interface bond2 bond mlag id 2
cumulus@leaf01:mgmt:~$ nv set interface bond1,bond2 link mtu 9000
cumulus@leaf01:mgmt:~$ nv set bridge domain br_default
cumulus@leaf01:mgmt:~$ nv set bridge domain br_default vlan 10,20,30
cumulus@leaf01:mgmt:~$ nv set interface bond1,bond2 bridge domain br_default
cumulus@leaf01:mgmt:~$ nv set interface bond1,bond2 bridge domain br_default vlan 10,20,30
cumulus@leaf01:mgmt:~$ nv config apply
cumulus@leaf01:mgmt:~$ nv config save
cumulus@leaf02:mgmt:~$ nv set interface peerlink bond member swp2-3
cumulus@leaf02:mgmt:~$ nv set mlag peer-ip linklocal
cumulus@leaf02:mgmt:~$ nv set mlag backup 192.168.200.11 vrf mgmt
cumulus@leaf02:mgmt:~$ nv set mlag mac-address 44:38:39:BE:EF:AA
cumulus@leaf02:mgmt:~$ nv set mlag init-delay 10
cumulus@leaf02:mgmt:~$ nv set interface bond1 bond member swp1
cumulus@leaf02:mgmt:~$ nv set interface bond2 bond member swp4-swp5
cumulus@leaf02:mgmt:~$ nv set interface bond1 bond mlag id 1
cumulus@leaf02:mgmt:~$ nv set interface bond2 bond mlag id 2
cumulus@leaf02:mgmt:~$ nv set interface bond1,bond2 link mtu 9000
cumulus@leaf02:mgmt:~$ nv set bridge domain br_default
cumulus@leaf02:mgmt:~$ nv set bridge domain br_default vlan 10,20,30
cumulus@leaf02:mgmt:~$ nv set interface bond1,bond2 bridge domain br_default
cumulus@leaf02:mgmt:~$ nv set interface bond1,bond2 bridge domain br_default vlan 10,20,30
cumulus@leaf02:mgmt:~$ nv config apply
cumulus@leaf02:mgmt:~$ nv config save
cumulus@spine01:mgmt:~$ nv set interface peerlink bond member swp3-4
cumulus@spine01:mgmt:~$ nv set mlag peer-ip linklocal
cumulus@spine01:mgmt:~$ nv set mlag backup 192.168.200.14 vrf mgmt
cumulus@spine01:mgmt:~$ nv set mlag mac-address 44:38:39:BE:EF:BB
cumulus@spine01:mgmt:~$ nv set mlag priority 1000
cumulus@spine01:mgmt:~$ nv set mlag init-delay 10
cumulus@spine01:mgmt:~$ nv set interface bond3 bond member swp1-2
cumulus@spine01:mgmt:~$ nv set interface bond3 bond mlag id 3
cumulus@spine01:mgmt:~$ nv set interface bond3 link mtu 9000
cumulus@spine01:mgmt:~$ nv set bridge domain br_default
cumulus@spine01:mgmt:~$ nv set bridge domain br_default vlan 10,20,30
cumulus@spine01:mgmt:~$ nv set interface bond3 bridge domain br_default
cumulus@spine01:mgmt:~$ nv set interface bond3 bridge domain br_default vlan 10,20,30
cumulus@spine01:mgmt:~$ nv set interface vlan10 ip address 10.1.10.2/24
cumulus@spine01:mgmt:~$ nv set interface vlan10 ip ip vrr address 10.1.10.1/24
cumulus@spine01:mgmt:~$ nv set interface vlan10 ip vrr mac-address 00:00:00:00:00:10
cumulus@spine01:mgmt:~$ nv set interface vlan10 ip vrr state up
cumulus@spine01:mgmt:~$ nv set interface vlan20 ip address 10.1.20.2/24
cumulus@spine01:mgmt:~$ nv set interface vlan20 ip ip vrr address 10.1.20.1/24
cumulus@spine01:mgmt:~$ nv set interface vlan20 ip vrr mac-address 00:00:00:00:00:20
cumulus@spine01:mgmt:~$ nv set interface vlan20 ip vrr state up
cumulus@spine01:mgmt:~$ nv set interface vlan30 ip address 10.1.30.2/24
cumulus@spine01:mgmt:~$ nv set interface vlan30 ip ip vrr address 10.1.30.1/24
cumulus@spine01:mgmt:~$ nv set interface vlan30 ip vrr mac-address 00:00:00:00:00:30
cumulus@spine01:mgmt:~$ nv set interface vlan30 ip vrr state up
cumulus@spine01:mgmt:~$ nv config apply
cumulus@spine01:mgmt:~$ nv config save
cumulus@spine02:mgmt:~$ nv set interface peerlink bond member swp3-4
cumulus@spine02:mgmt:~$ nv set mlag peer-ip linklocal
cumulus@spine02:mgmt:~$ nv set mlag backup 192.168.200.13 vrf mgmt
cumulus@spine02:mgmt:~$ nv set mlag mac-address 44:38:39:BE:EF:BB
cumulus@spine02:mgmt:~$ nv set mlag init-delay 10
cumulus@spine02:mgmt:~$ nv set interface bond3 bond member swp1-2
cumulus@spine02:mgmt:~$ nv set interface bond3 bond mlag id 3
cumulus@spine02:mgmt:~$ nv set interface bond3 link mtu 9000
cumulus@spine02:mgmt:~$ nv set bridge domain br_default
cumulus@spine02:mgmt:~$ nv set bridge domain br_default vlan 10,20,30
cumulus@spine02:mgmt:~$ nv set interface bond3 bridge domain br_default
cumulus@spine02:mgmt:~$ nv set interface bond3 bridge domain br_default vlan 10,20,30
cumulus@spine02:mgmt:~$ nv set interface vlan10 ip address 10.1.10.3/24
cumulus@spine02:mgmt:~$ nv set interface vlan10 ip ip vrr address 10.1.10.1/24
cumulus@spine02:mgmt:~$ nv set interface vlan10 ip vrr mac-address 00:00:00:00:00:10
cumulus@spine02:mgmt:~$ nv set interface vlan10 ip vrr state up
cumulus@spine02:mgmt:~$ nv set interface vlan20 ip address 10.1.20.3/24
cumulus@spine02:mgmt:~$ nv set interface vlan20 ip ip vrr address 10.1.20.1/24
cumulus@spine02:mgmt:~$ nv set interface vlan20 ip vrr mac-address 00:00:00:00:00:20
cumulus@spine02:mgmt:~$ nv set interface vlan20 ip vrr state up
cumulus@spine02:mgmt:~$ nv set interface vlan30 ip address 10.1.30.3/24
cumulus@spine02:mgmt:~$ nv set interface vlan30 ip ip vrr address 10.1.30.1/24
cumulus@spine02:mgmt:~$ nv set interface vlan30 ip vrr mac-address 00:00:00:00:00:30
cumulus@spine02:mgmt:~$ nv set interface vlan30 ip vrr state up
cumulus@spine02:mgmt:~$ nv config apply
cumulus@spine02:mgmt:~$ nv config save
cumulus@leaf01:mgmt:~$ nv set interface peerlink bond member swp2-3
cumulus@leaf01:mgmt:~$ nv set mlag peer-ip linklocal
cumulus@leaf01:mgmt:~$ nv set mlag backup 192.168.200.12 vrf mgmt
cumulus@leaf01:mgmt:~$ nv set mlag mac-address 44:38:39:BE:EF:AA
cumulus@leaf01:mgmt:~$ nv set mlag priority 1000
cumulus@leaf01:mgmt:~$ nv set mlag init-delay 10
cumulus@leaf01:mgmt:~$ nv set interface bond2 bond member swp4-swp5
cumulus@leaf01:mgmt:~$ nv set interface bond2 bond mlag id 2
cumulus@leaf01:mgmt:~$ nv set interface swp1,bond2 link mtu 9000
cumulus@leaf01:mgmt:~$ nv set bridge domain br_default
cumulus@leaf01:mgmt:~$ nv set bridge domain br_default vlan 10,20,30
cumulus@leaf01:mgmt:~$ nv set interface swp1,bond2 bridge domain br_default
cumulus@leaf01:mgmt:~$ nv set interface swp1,bond2 bridge domain br_default vlan 10,20,30
cumulus@leaf01:mgmt:~$ nv config apply
cumulus@leaf01:mgmt:~$ nv config save
cumulus@leaf02:mgmt:~$ nv set interface peerlink bond member swp2-3
cumulus@leaf02:mgmt:~$ nv set mlag peer-ip linklocal
cumulus@leaf02:mgmt:~$ nv set mlag backup 192.168.200.11 vrf mgmt
cumulus@leaf02:mgmt:~$ nv set mlag mac-address 44:38:39:BE:EF:AA
cumulus@leaf02:mgmt:~$ nv set mlag init-delay 10
cumulus@leaf02:mgmt:~$ nv set interface bond2 bond member swp4-swp5
cumulus@leaf02:mgmt:~$ nv set interface bond2 bond mlag id 2
cumulus@leaf02:mgmt:~$ nv set interface swp1,bond2 link mtu 9000
cumulus@leaf02:mgmt:~$ nv set bridge domain br_default
cumulus@leaf02:mgmt:~$ nv set bridge domain br_default vlan 10,20,30
cumulus@leaf02:mgmt:~$ nv set interface swp1,bond2 bridge domain br_default
cumulus@leaf02:mgmt:~$ nv set interface swp1,bond2 bridge domain br_default vlan 10,20,30
cumulus@leaf02:mgmt:~$ nv config apply
cumulus@leaf02:mgmt:~$ nv config save
cumulus@spine01:mgmt:~$ nv set interface peerlink bond member swp3-4
cumulus@spine01:mgmt:~$ nv set mlag peer-ip linklocal
cumulus@spine01:mgmt:~$ nv set mlag backup 192.168.200.14 vrf mgmt
cumulus@spine01:mgmt:~$ nv set mlag mac-address 44:38:39:BE:EF:BB
cumulus@spine01:mgmt:~$ nv set mlag priority 1000
cumulus@spine01:mgmt:~$ nv set mlag init-delay 10
cumulus@spine01:mgmt:~$ nv set interface bond3 bond member swp1-2
cumulus@spine01:mgmt:~$ nv set interface bond3 bond mlag id 3
cumulus@spine01:mgmt:~$ nv set interface bond3 link mtu 9000
cumulus@spine01:mgmt:~$ nv set bridge domain br_default
cumulus@spine01:mgmt:~$ nv set bridge domain br_default vlan 10,20,30
cumulus@spine01:mgmt:~$ nv set interface bond3 bridge domain br_default
cumulus@spine01:mgmt:~$ nv set interface bond3 bridge domain br_default vlan 10,20,30
cumulus@spine01:mgmt:~$ nv set interface vlan10 ip address 10.1.10.2/24
cumulus@spine01:mgmt:~$ nv set interface vlan10 ip ip vrr address 10.1.10.1/24
cumulus@spine01:mgmt:~$ nv set interface vlan10 ip vrr mac-address 00:00:00:00:00:10
cumulus@spine01:mgmt:~$ nv set interface vlan10 ip vrr state up
cumulus@spine01:mgmt:~$ nv set interface vlan20 ip address 10.1.20.2/24
cumulus@spine01:mgmt:~$ nv set interface vlan20 ip ip vrr address 10.1.20.1/24
cumulus@spine01:mgmt:~$ nv set interface vlan20 ip vrr mac-address 00:00:00:00:00:20
cumulus@spine01:mgmt:~$ nv set interface vlan20 ip vrr state up
cumulus@spine01:mgmt:~$ nv set interface vlan30 ip address 10.1.30.2/24
cumulus@spine01:mgmt:~$ nv set interface vlan30 ip ip vrr address 10.1.30.1/24
cumulus@spine01:mgmt:~$ nv set interface vlan30 ip vrr mac-address 00:00:00:00:00:30
cumulus@spine01:mgmt:~$ nv set interface vlan30 ip vrr state up
cumulus@spine01:mgmt:~$ nv config apply
cumulus@spine01:mgmt:~$ nv config save
cumulus@spine02:mgmt:~$ nv set interface peerlink bond member swp3-4
cumulus@spine02:mgmt:~$ nv set mlag peer-ip linklocal
cumulus@spine02:mgmt:~$ nv set mlag backup 192.168.200.13 vrf mgmt
cumulus@spine02:mgmt:~$ nv set mlag mac-address 44:38:39:BE:EF:BB
cumulus@spine02:mgmt:~$ nv set mlag init-delay 10
cumulus@spine02:mgmt:~$ nv set interface bond3 bond member swp1-2
cumulus@spine02:mgmt:~$ nv set interface bond3 bond mlag id 3
cumulus@spine02:mgmt:~$ nv set interface bond3 link mtu 9000
cumulus@spine02:mgmt:~$ nv set bridge domain br_default
cumulus@spine02:mgmt:~$ nv set bridge domain br_default vlan 10,20,30
cumulus@spine02:mgmt:~$ nv set interface bond3 bridge domain br_default
cumulus@spine02:mgmt:~$ nv set interface bond3 bridge domain br_default vlan 10,20,30
cumulus@spine02:mgmt:~$ nv set interface vlan10 ip address 10.1.10.3/24
cumulus@spine02:mgmt:~$ nv set interface vlan10 ip ip vrr address 10.1.10.1/24
cumulus@spine02:mgmt:~$ nv set interface vlan10 ip vrr mac-address 00:00:00:00:00:10
cumulus@spine02:mgmt:~$ nv set interface vlan10 ip vrr state up
cumulus@spine02:mgmt:~$ nv set interface vlan20 ip address 10.1.20.3/24
cumulus@spine02:mgmt:~$ nv set interface vlan20 ip ip vrr address 10.1.20.1/24
cumulus@spine02:mgmt:~$ nv set interface vlan20 ip vrr mac-address 00:00:00:00:00:20
cumulus@spine02:mgmt:~$ nv set interface vlan20 ip vrr state up
cumulus@spine02:mgmt:~$ nv set interface vlan30 ip address 10.1.30.3/24
cumulus@spine02:mgmt:~$ nv set interface vlan30 ip ip vrr address 10.1.30.1/24
cumulus@spine02:mgmt:~$ nv set interface vlan30 ip vrr mac-address 00:00:00:00:00:30
cumulus@spine02:mgmt:~$ nv set interface vlan30 ip vrr state up
cumulus@spine02:mgmt:~$ nv config apply
cumulus@spine02:mgmt:~$ nv config save
配置验证
您可以使用 NVUE 或 Linux 命令验证配置。
MLAG
cumulus@leaf01:mgmt:~$ nv show mlag
operational applied description
-------------- ----------------------- ----------------- ------------------------------------------------------
enable on Turn the feature 'on' or 'off'. The default is 'off'.
debug off Enable MLAG debugging
init-delay 10 The delay, in seconds, before bonds are brought up.
mac-address 44:38:39:be:ef:aa 44:38:39:BE:EF:AA Override anycast-mac and anycast-id
peer-ip fe80::4638:39ff:fe00:22 linklocal Peer Ip Address
priority 1000 1000 Mlag Priority
[backup] 192.168.200.12 192.168.200.12 Set of MLAG backups
backup-active True Mlag Backup Status
backup-reason Mlag Backup Reason
local-id 44:38:39:00:00:21 Mlag Local Unique Id
local-role primary Mlag Local Role
peer-alive True Mlag Peer Alive Status
peer-id 44:38:39:00:00:22 Mlag Peer Unique Id
peer-interface peerlink.4094 Mlag Peerlink Interface
peer-priority 32768 Mlag Peer Priority
peer-role secondary Mlag Peer Role
cumulus@leaf02:mgmt:~$ nv show mlag
operational applied description
-------------- ----------------------- ----------------- ------------------------------------------------------
enable on Turn the feature 'on' or 'off'. The default is 'off'.
debug off Enable MLAG debugging
init-delay 10 The delay, in seconds, before bonds are brought up.
mac-address 44:38:39:be:ef:aa 44:38:39:BE:EF:AA Override anycast-mac and anycast-id
peer-ip fe80::4638:39ff:fe00:21 linklocal Peer Ip Address
priority 32768 32768 Mlag Priority
[backup] 192.168.200.11 192.168.200.11 Set of MLAG backups
backup-active True Mlag Backup Status
backup-reason Mlag Backup Reason
local-id 44:38:39:00:00:22 Mlag Local Unique Id
local-role secondary Mlag Local Role
peer-alive True Mlag Peer Alive Status
peer-id 44:38:39:00:00:21 Mlag Peer Unique Id
peer-interface peerlink.4094 Mlag Peerlink Interface
peer-priority 1000 Mlag Peer Priority
peer-role primary Mlag Peer Role
cumulus@leaf01:mgmt:~$ clagctl
The peer is alive
Our Priority, ID, and Role: 1000 44:38:39:00:00:21 primary
Peer Priority, ID, and Role: 32768 44:38:39:00:00:22 secondary
Peer Interface and IP: peerlink.4094 fe80::4638:39ff:fe00:22 (linklocal)
Backup IP: 192.168.200.12 (active)
System MAC: 44:38:39:be:ef:aa
CLAG Interfaces
Our Interface Peer Interface CLAG Id Conflicts Proto-Down Reason
---------------- ---------------- ------- -------------------- -----------------
bond1 bond1 1 - -
cumulus@leaf02:mgmt:~$ clagctl
The peer is alive
Our Priority, ID, and Role: 32768 44:38:39:00:00:22 secondary
Peer Priority, ID, and Role: 1000 44:38:39:00:00:21 primary
Peer Interface and IP: peerlink.4094 fe80::4638:39ff:fe00:21 (linklocal)
Backup IP: 192.168.200.11 (active)
System MAC: 44:38:39:be:ef:aa
CLAG Interfaces
Our Interface Peer Interface CLAG Id Conflicts Proto-Down Reason
---------------- ---------------- ------- -------------------- -----------------
bond1 bond1 1 - -
网桥和 VLAN
cumulus@leaf01:mgmt:~$ nv show bridge domain br_default
operational applied description
------------- ----------- ---------- ----------------------------------------------------------------------
encap 802.1Q 802.1Q Interfaces added to this domain will, by default, use this encapsul...
mac-address auto Override global mac address
type vlan-aware vlan-aware Type of bridge domain.
untagged 1 Interfaces added to this domain will, by default, be trunk interfac...
multicast
snooping
enable on off Turn the feature 'on' or 'off'. The default is 'off'.
querier
enable off Turn the feature 'on' or 'off'. The default is 'off'.
stp
priority 32768 32768 stp priority. The priority value must be a number between 4096 and...
state up up The state of STP on the bridge
[vlan] 10 Set of vlans in the bridge domain. Only applicable when the domain...
[vlan] 20
[vlan] 30
[mdb] Set of mdb entries in the bridge domain
[router-port] 1 Set of multicast router ports
cumulus@leaf02:mgmt:~$ nv show bridge domain br_default
operational applied description
------------- ----------- ---------- ----------------------------------------------------------------------
encap 802.1Q 802.1Q Interfaces added to this domain will, by default, use this encapsul...
mac-address auto Override global mac address
type vlan-aware vlan-aware Type of bridge domain.
untagged 1 Interfaces added to this domain will, by default, be trunk interfac...
multicast
snooping
enable on off Turn the feature 'on' or 'off'. The default is 'off'.
querier
enable off Turn the feature 'on' or 'off'. The default is 'off'.
stp
priority 32768 32768 stp priority. The priority value must be a number between 4096 and...
state up up The state of STP on the bridge
[vlan] 10 Set of vlans in the bridge domain. Only applicable when the domain...
[vlan] 20
[vlan] 30
[mdb] Set of mdb entries in the bridge domain
[router-port] 1 Set of multicast router ports
cumulus@leaf01:mgmt:~$ bridge vlan show
port vlan ids
peerlink 1 PVID Egress Untagged
10
20
30
bond1 1 PVID Egress Untagged
10
20
30
br_default 10
20
30
cumulus@leaf02:mgmt:~$ bridge vlan show
port vlan ids
peerlink 1 PVID Egress Untagged
10
20
30
bond1 1 PVID Egress Untagged
10
20
30
br_default 10
20
30
接口、SVI 和 VRR
cumulus@leaf01:mgmt:~$ nv show interface
Interface MTU Speed State Remote Host Remote Port Type Summary
--------------- ----- ----- ----- --------------- ----------------- -------- -----------------------------
+ bond1 9000 1G up bond
+ eth0 1500 1G up oob-mgmt-switch swp6 eth IP Address: 192.168.200.11/24
+ lo 65536 up loopback IP Address: 10.10.10.1/32
lo IP Address: 127.0.0.1/8
lo IP Address: ::1/128
+ peerlink 9216 2G up bond
+ peerlink.4094 9216 up sub
+ swp1 9000 1G up server01 44:38:39:00:00:12 swp
+ swp2 9216 1G up leaf02 swp2 swp
+ swp3 9216 1G up leaf02 swp3 swp
+ swp4 9216 1G up spine01 swp1 swp
+ swp5 9216 1G up spine02 swp1 swp
+ vlan10 9216 up svi IP Address: 10.1.10.2/24
+ vlan20 9216 up svi IP Address: 10.1.20.2/24
+ vlan30 9216 up svi IP Address: 10.1.30.2/24
cumulus@leaf02:mgmt:~$ nv show interface
Interface MTU Speed State Remote Host Remote Port Type Summary
--------------- ----- ----- ----- --------------- ----------------- -------- -----------------------------
+ bond1 9000 1G up bond
+ eth0 1500 1G up oob-mgmt-switch swp6 eth IP Address: 192.168.200.12/24
+ lo 65536 up loopback IP Address: 10.10.10.2/32
lo IP Address: 127.0.0.1/8
lo IP Address: ::1/128
+ peerlink 9216 2G up bond
+ peerlink.4094 9216 up sub
+ swp1 9000 1G up server01 44:38:39:00:00:16 swp
+ swp2 9216 1G up leaf01 swp2 swp
+ swp3 9216 1G up leaf01 swp3 swp
+ swp4 9216 1G up spine01 swp2 swp
+ swp5 9216 1G up spine02 swp2 swp
+ vlan10 9216 up svi IP Address: 10.1.10.3/24
+ vlan20 9216 up svi IP Address: 10.1.20.3/24
+ vlan30 9216 up svi IP Address: 10.1.30.3/24
cumulus@leaf01:mgmt:~$ nv show interface vlan10 ip vrr
operational applied description
----------- ----------------- ----------------- ------------------------------------------------------
enable on Turn the feature 'on' or 'off'. The default is 'off'.
mac-address 00:00:00:00:00:10 00:00:00:00:00:10 Override anycast-mac
mac-id none Override anycast-id
[address] 10.1.10.1/24 10.1.10.1/24 Virtual addresses with prefixes
state up up The state of the interface
cumulus@leaf01:mgmt:~$ nv show interface vlan20 ip vrr
operational applied description
----------- ----------------- ----------------- ------------------------------------------------------
enable on Turn the feature 'on' or 'off'. The default is 'off'.
mac-address 00:00:00:00:00:20 00:00:00:00:00:20 Override anycast-mac
mac-id none Override anycast-id
[address] 10.1.20.1/24 10.1.20.1/24 Virtual addresses with prefixes
state up up The state of the interface
cumulus@leaf01:mgmt:~$ nv show interface vlan30 ip vrr
operational applied description
----------- ----------------- ----------------- ------------------------------------------------------
enable on Turn the feature 'on' or 'off'. The default is 'off'.
mac-address 00:00:00:00:00:30 00:00:00:00:00:30 Override anycast-mac
mac-id none Override anycast-id
[address] 10.1.30.1/24 10.1.30.1/24 Virtual addresses with prefixes
state up up The state of the interface
cumulus@leaf02:mgmt:~$ nv show interface vlan10 ip vrr
operational applied description
----------- ----------------- ----------------- ------------------------------------------------------
enable on Turn the feature 'on' or 'off'. The default is 'off'.
mac-address 00:00:00:00:00:10 00:00:00:00:00:10 Override anycast-mac
mac-id none Override anycast-id
[address] 10.1.10.1/24 10.1.10.1/24 Virtual addresses with prefixes
state up up The state of the interface
cumulus@leaf02:mgmt:~$ nv show interface vlan20 ip vrr
operational applied description
----------- ----------------- ----------------- ------------------------------------------------------
enable on Turn the feature 'on' or 'off'. The default is 'off'.
mac-address 00:00:00:00:00:20 00:00:00:00:00:20 Override anycast-mac
mac-id none Override anycast-id
[address] 10.1.20.1/24 10.1.20.1/24 Virtual addresses with prefixes
state up up The state of the interface
cumulus@leaf02:mgmt:~$ nv show interface vlan30 ip vrr
operational applied description
----------- ----------------- ----------------- ------------------------------------------------------
enable on Turn the feature 'on' or 'off'. The default is 'off'.
mac-address 00:00:00:00:00:30 00:00:00:00:00:30 Override anycast-mac
mac-id none Override anycast-id
[address] 10.1.30.1/24 10.1.30.1/24 Virtual addresses with prefixes
state up up The state of the interface
在三层 Fabric 上扩展二层域（VXLAN 和 EVPN）
如果您的 ESF 环境在二层域中需要两个以上的 Spine 交换机，或者您需要通过三层 Fabric 隧道传输二层存储流量，请使用 VXLAN 双活模式 模式和 EVPN。在此部署中，您必须使用 BGP 协议才能使用 EVPN 控制平面。
如果您的 ESF 环境仅使用单个 VLAN 或 VLAN 不通信，则需要设置 二层扩展 (Layer 2 stretch) 功能。
如果您需要在 ESF 部署中部署 EVPN 子网间路由，Cumulus Linux 提供两个选项
- 集中式路由，其中指定的 VTEP 充当子网的三层网关，并在它们之间执行路由。其余 VTEP 仅执行桥接。
- 分布式对称路由，其中所有 VTEP 都在入口和出口执行路由。
在使用 EVPN 网络时，您可以利用 EVPN 多宿主 (EVPN MH) 来消除环境中对 MLAG 的需求。通过从 ToR 中删除 MLAG，您可以使用更大的二层冗余组（两个以上的 ToR）。EVPN 多宿主使用标准 BGP-EVPN 控制平面，并允许跨供应商互操作性，以便您可以部署多供应商 ToR。
VXLAN 和 EVPN 以及 EVPN MH 在 ESF 网络中是罕见的配置，因此本指南不包含逐步配置。您可以在 此处 找到一些配置示例。
查看 网络虚拟化 和 EVPN 多宿主 文档，了解更多详细信息和配置命令。
如果您使用 NSX-T 环境，请查看此 Cumulus Linux VMware NSX-T 部署指南。
基于融合以太网的 RDMA (RoCE)
当 ESF 运行 RoCE 流量时，您必须配置流量路径中的所有网络元素以处理 RoCE。Cumulus Linux 提供了一个 RoCE 命令，该命令会自动在交换机上设置所有 RoCE 最佳实践配置。该命令设置实现最佳性能所需的所有设置。此配置包括缓冲池设置、流量分类和优先级映射、PFC 和 ECN 阈值以及常规、RoCE 和 CNP 数据包的排队和调度设置。
要配置交换机以使用 RoCE
cumulus@switch:mgmt:~$ nv set qos roce
Cumulus Linux 支持 RoCEv1 和 RoCEv2 的单个命令配置。默认 RoCE 模式为无损模式，具有 PFC 和 ECN。您可以设置有损模式以仅使用 ECN。
要配置 RoCE 有损模式
cumulus@switch:mgmt:~$ nv set qos roce lossy
有关 Cumulus Linux RoCE 默认最佳实践配置的更多信息，请参阅 基于融合以太网的 RDMA - RoCE。
配置管理
Cumulus Linux 具有声明式 CLI，具有提交-确认和配置管理功能。所有配置命令仅在您应用它们后才生效，并在您保存它们后保持持久。
要将配置作为运行配置应用
cumulus@switch:mgmt:~$ nv config apply
要将配置另存为持久启动配置
cumulus@switch:mgmt:~$ nv config save
保存配置后，NVUE 会创建一个 startup.yaml 配置文件。Cumulus Linux 使用此 YAML 文件作为交换机上的启动配置。
NVUE 有许多有用的配置管理选项。查看 NVUE 文档的 配置管理命令 部分以了解更多详细信息。