头节点配置#
本节介绍在 Base Command Manager 头节点上执行的配置步骤。
在 /cm/local/apps/cmd/etc/cmd.conf 中，取消注释 AdvancedConfig 参数并将其设置为以下值。
AdvancedConfig = { "DeviceResolveAnyMAC=1" } # modified value
重启 CMDaemon 以启用从绑定接口的可靠 PXE 启动。
systemctl restart cmd
创建一个新文件
/cm/local/apps/cmd/etc/htdocs/disk-setup/dgx-disksetup.xml，内容如下。<?xml version="1.0" encoding="UTF-8"?> <diskSetup> <device> <blockdev>/dev/nvme0n1</blockdev> <partition id="boot1" partitiontype="esp"> <size>512M</size> <type>linux</type> <filesystem>fat</filesystem> <mountPoint>/boot/efi</mountPoint> <mountOptions>defaults,noatime,nodiratime</mountOptions> </partition> <partition id="slash1"> <size>max</size> <type>linux raid</type> </partition> </device> <device> <blockdev>/dev/nvme1n1</blockdev> <partition id="boot2" partitiontype="esp"> <size>512M</size> <type>linux</type> <filesystem>fat</filesystem> <mountOptions>defaults,noatime,nodiratime</mountOptions> </partition> <partition id="slash2"> <size>max</size> <type>linux raid</type> </partition> </device> <device> <blockdev>/dev/nvme2n1</blockdev> <partition id="raid1" partitiontype="esp"> <size>max</size> <type>linux raid</type> </partition> </device> <device> <blockdev>/dev/nvme3n1</blockdev> <partition id="raid2" partitiontype="esp"> <size>max</size> <type>linux raid</type> </partition> </device> <device> <blockdev>/dev/nvme4n1</blockdev> <partition id="raid3" partitiontype="esp"> <size>max</size> <type>linux raid</type> </partition> </device> <device> <blockdev>/dev/nvme5n1</blockdev> <partition id="raid4" partitiontype="esp"> <size>max</size> <type>linux raid</type> </partition> </device> <device> <blockdev>/dev/nvme6n1</blockdev> <partition id="raid5" partitiontype="esp"> <size>max</size> <type>linux raid</type> </partition> </device> <device> <blockdev>/dev/nvme7n1</blockdev> <partition id="raid6" partitiontype="esp"> <size>max</size> <type>linux raid</type> </partition> </device> <device> <blockdev>/dev/nvme8n1</blockdev> <partition id="raid7" partitiontype="esp"> <size>max</size> <type>linux raid</type> </partition> </device> <device> <blockdev>/dev/nvme9n1</blockdev> <partition id="raid8" partitiontype="esp"> <size>max</size> <type>linux raid</type> </partition> </device> <raid id="slash"> <member>slash1</member> <member>slash2</member> <level>1</level> <filesystem>ext4</filesystem> <mountPoint>/</mountPoint> <mountOptions>defaults,noatime,nodiratime</mountOptions> </raid> <raid id="raid"> <member>raid1</member> <member>raid2</member> <member>raid3</member> <member>raid4</member> <member>raid5</member> <member>raid6</member> <member>raid7</member> <member>raid8</member> <level>0</level> <filesystem>ext4</filesystem> <mountPoint>/raid</mountPoint> <mountOptions>defaults,noatime,nodiratime</mountOptions> </raid> </diskSetup>
其余的头节点配置将从 cmsh 工具中完成。在 cmsh 中的任何菜单中输入 help 将显示有效命令及其简短描述。
添加并配置 Base Command Manager 的网络。从 ipminet 开始，它将用于连接到所有节点的 BMC。确保这些值与现场勘测一致。
cmsh [bcm10-headnode]% network [bcm10-headnode->network]% add ipminet [bcm10-headnode->network*[ipminet*]]% set netmaskbits 24 [bcm10-headnode->network*[ipminet*]]% set baseaddress 10.133.10.0 [bcm10-headnode->network*[ipminet*]]% set domainname cm.ipmi [bcm10-headnode->network*[ipminet*]]% set excludefromsearchdomain yes [bcm10-headnode->network*[ipminet*]]% set disableautomaticexports yes [bcm10-headnode->network*[ipminet*]]% set gateway 10.133.10.1 [bcm10-headnode->network*[ipminet*]]% commit
现在创建 dgxnet。此网络将用于 DGX 系统的带内管理。由于它与 internalnet 相似，克隆它将节省时间。
cmsh [bcm10-headnode]% network [bcm10-headnode->network]% clone internalnet dgxnet [bcm10-headnode->network*[dgxnet*]]% set domainname cm.dgx [bcm10-headnode->network*[dgxnet*]]% set baseaddress 10.133.15.0 [bcm10-headnode->network*[dgxnet*]]% set gateway 10.133.15.1
接下来为存储和计算 fabric 创建 ib 网络。
cmsh [bcm10-headnode]% network [bcm10-headnode->network*[computenet*]]% set domainname ib.compute [bcm10-headnode->network*[computenet*]]% set disableautomaticexports yes [bcm10-headnode->network*[computenet*]]% set baseaddress 100.126.0.0 [bcm10-headnode->network*[computenet*]]% set netmaskbits 16 [bcm10-headnode->network*[computenet*]]% commit [bcm10-headnode->network[computenet]]% clone computenet storagenet [bcm10-headnode->network*[storagenet*]]% set domainname ib.storage [bcm10-headnode->network*[storagenet*]]% set baseaddress 100.127.0.0 [bcm10-headnode->network*[storagenet*]]% commit
为 DGX 节点创建备份镜像。
cmsh [bcm10-headnode]% softwareimage [bcm10-headnode->softwareimage]% clone dgx-os-6.2-h100-image dgx-os-6.2-h100-image-orig [bcm10-headnode->softwareimage*[dgx-os-6.2-h100-image-orig*]]% commit
配置 DGX 节点的类别。
cmsh [bcm10-headnode]% category [bcm10-headnode->category]% use dgx-h100 [bcm10-headnode->category[dgx-h100]]% set managementnetwork dgxnet [bcm10-headnode->category*[dgx-h100*]]% set bootloader grub [bcm10-headnode->category*[dgx-h100*]]% set disksetup /cm/local/apps/cmd/etc/htdocs/disk-setup/dgx-disksetup.xml [bcm10-headnode->category[dgx-h100]]% commit # TODO - copy the disk setup config from parker and set it here.
接下来，将集群中的 DGX 节点添加到 cmsh。
从 dgx-01 开始。
cmsh [bcm10-headnode]% device [bcm10-headnode->device]% clone node001 dgx-01 [bcm10-headnode->device*[dgx-01*]]% set category dgx-h100 [bcm10-headnode->device*[dgx-01*]]% interfaces [bcm10-headnode->device*[dgx-01*]->interfaces]% add bmc ipmi0 10.133.10.31 ipminet [bcm10-headnode->device*[dgx-01*]->interfaces*[ipmi0*]]% add physical enp170s0f1np1; add physical enp41s0f1np1 [bcm10-headnode->device*[dgx-01*]->interfaces*[enp41s0f1np1*]]% add bond bond0 10.133.15.31 dgxnet [bcm10-headnode->device*[dgx-01*]->interfaces*[bond0*]]% append interfaces enp170s0f1np1 enp41s0f1np1 [bcm10-headnode->device*[dgx-01*]->interfaces*[bond0*]]% .. [bcm10-headnode->device*[dgx-01*]->interfaces*]% remove bootif [bcm10-headnode->device*[dgx-01*]->interfaces*]% .. [bcm10-headnode->device*[dgx-01*]]% set provisioninginterface bond0 [bcm10-headnode->device*[dgx-01*]]% interfaces [bcm10-headnode->device*[dgx-01*]]% commit
现在添加 ib 接口。
[bcm10-headnode->device*[dgx-01*]->interfaces[bond0]]% add physical ibp154s0 100.126.0.31 computenet [bcm10-headnode->device*[dgx-01*]->interfaces*[ibp154s0*]]% foreach -o ibp154s0 ibp192s0 ibp206s0 ibp220s0 ibp24s0 ibp64s0 ibp79s0 ibp94s0 () [bcm10-headnode->device*[dgx-01*]->interfaces*]% set ibp192s0 ip 100.126.1.31 [bcm10-headnode->device*[dgx-01*]->interfaces*]% set ibp206s0 ip 100.126.2.31 [bcm10-headnode->device*[dgx-01*]->interfaces*]% set ibp220s0 ip 100.126.3.31 [bcm10-headnode->device*[dgx-01*]->interfaces*]% set ibp24s0 ip 100.126.4.31 [bcm10-headnode->device*[dgx-01*]->interfaces*]% set ibp64s0 ip 100.126.5.31 [bcm10-headnode->device*[dgx-01*]->interfaces*]% set ibp79s0 ip 100.126.6.31 [bcm10-headnode->device*[dgx-01*]->interfaces*]% set ibp94s0 ip 100.126.7.31 [bcm10-headnode->device*[dgx-01*]->interfaces*]% commit [bcm10-headnode->device[dgx-01]->interfaces]% add physical ibp170s0f0 100.127.0.31 storagenet [bcm10-headnode->device*[dgx-01*]->interfaces*[ibp170s0f0*]]% add physical ibp41s0f0 100.127.1.31 storagenet [bcm10-headnode->device*[dgx-01*]->interfaces*[ibp41s0f0*]]% commit
设置带内管理接口的 mac 地址并设置 provisioning 接口。系统的 MAC 地址可以是以下接口使用的任何物理 MAC 地址。
注意
仔细检查每个接口的 MAC 地址和 bond0 接口的 IP 地址。这里的错误将难以诊断。
[bcm10-headnode->device[dgx-01]->interfaces[ibp41s0f0]]% exit [bcm10-headnode->device[dgx-01]->interfaces]% set enp170s0f1np1 mac 94:6D:00:00:00:FB [bcm10-headnode->device*[dgx-01*]->interfaces*]% set enp41s0f1np1 mac 94:00:00:00:74:0B [bcm10-headnode->device*[dgx-01*]->interfaces*]% exit [bcm10-headnode->device*[dgx-01*]]% set mac 94:6D:00:00:00:FB [bcm10-headnode->device*[dgx-01*]]% commit
DGX-01 现在可以进行配置。必须直接或使用 KVM 开机。此过程将花费几分钟才能完成 BIOS。之后，当节点正在配置时，将显示节点状态进度。监控 /var/log/messages 和 /var/log/node-installer 日志文件以验证一切是否顺利进行。
如果 dgx-01 的状态变为 UP，则继续启动其余的 DGX 节点。
[bcm10-headnode->device[dgx-01]]% ls Type Hostname (key) MAC Category Ip Network Status ---------------------- ---------------- ------------------ ---------------- --------------- -------------- ----------------------- HeadNode bcm10-headnode BC:00:00:00:43:45 10.133.11.51 internalnet [ UP ] PhysicalNode dgx-01 94:00:00:00:91:FB dgx-h100 10.133.15.31 dgxnet [ UP ] PhysicalNode node001 00:00:00:00:00:00 default 10.133.11.1 internalnet [ DOWN ], unassigned
克隆 dgx-01 以添加所有其他 DGX 节点。在本示例中，只有 4 个 DGX 节点。
[bcm10-headnode->device[dgx-01]]% foreach -o dgx-01 -n dgx-02..dgx-04 () --next-ip [bcm10-headnode->device*]% commit Successfully committed 3 Devices
为每个新节点设置 MAC 地址。为每个新的 DGX 节点重复以下步骤。
[bcm10-headnode->device]% use dgx-02; interfaces [bcm10-headnode->device[dgx-02]->interfaces]% set enp170s0f1np1 mac 94:6D:00:00:00:FD [bcm10-headnode->device*[dgx-02*]->interfaces*]% set enp41s0f1np1 mac 94:6D:00:00:00:FE [bcm10-headnode->device*[dgx-02*]->interfaces*]% exit [bcm10-headnode->device*[dgx-02*]]% set mac 94:6D:00:00:00:FD [bcm10-headnode->device*[dgx-02*]]% commit
开启电源并配置 DGX 节点。
（可选）如果头节点将使用绑定接口，请使用以下命令。您可能需要重启头节点并重做 request-license 步骤。
[bcm10-headnode]% device [bcm10-headnode->device]% use bcm10-headnode [bcm10-headnode->device[bcm10-headnode]]% interfaces [bcm10-headnode->device[bcm10-headnode]->interfaces]% clear ens3f1np1 ip [bcm10-headnode->device*[bcm10-headnode*]->interfaces*]% clear ens3f1np1 network [bcm10-headnode->device*[bcm10-headnode*]->interfaces*]% add physical ens2np0 [bcm10-headnode->device*[bcm10-headnode*]->interfaces*[ens2np0*]]% set mac 88:00:00:00:18:d8 [bcm10-headnode->device*[bcm10-headnode*]->interfaces*[ens2np0*]]% add bond bond0 [bcm10-headnode->device*[bcm10-headnode*]->interfaces*[bond0*]]% append interfaces ens3f1np1 ens2np0 [bcm10-headnode->device*[bcm10-headnode*]->interfaces*[bond0*]]% set mode 1 [bcm10-headnode->device*[bcm10-headnode*]->interfaces*[bond0*]]% set network internalnet [bcm10-headnode->device*[bcm10-headnode*]->interfaces*[bond0*]]% set ip 10.133.4.24 [bcm10-headnode->device*[bcm10-headnode*]->interfaces*[bond0*]]% .. [bcm10-headnode->device*[bcm10-headnode*]->interfaces*]% .. [bcm10-headnode->device*[bcm10-headnode*]]% set provisioninginterface bond0 [bcm10-headnode->device*[bcm10-headnode*]]% commit