cuBB 在 NVIDIA 云原生堆栈上#
NVIDIA 云原生堆栈（前身为 Cloud Native Core）是一系列在 NVIDIA GPU 上运行云原生工作负载的软件集合。本节介绍如何在 NVIDIA 云原生堆栈和相关组件上安装和运行 Aerial cuBB 软件示例，以运行 Aerial cuBB。
NVIDIA 云原生堆栈的安装#
先决条件：服务器必须已安装操作系统、NVIDIA 驱动程序以及其他配置，如在 Grace Hopper 上安装工具或在 Dell R750 上安装工具中所述。
安装 NVIDIA 云原生堆栈的步骤遵循 GitHub 上的 NVIDIA 云原生堆栈 v13.0 安装指南，从“安装容器运行时”部分开始，并附带以下补充说明
当安装指南中提供 containerd 或 CRI-O 选择时，请选择 containerd。
为了在没有 GPU 的服务器上运行 ru-emulator，有必要从该服务器上的
/etc/containerd/config.toml中删除/注释掉 “BinaryName” 字段。如果未执行此步骤，则可能会出现 ru-emulator 启动失败错误消息
State: Terminated Reason: StartError Message: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy' nvidia-container-cli: initialization error: nvml error: driver not loaded: unknown Exit Code: 128 Started: Thu, 01 Jan 1970 00:00:00 +0000 Finished: Wed, 17 Jan 2024 05:25:27 +0000
启用 k8s CPU Manager、Topology Manager 和 Memory Manager。
更新每个工作节点的 /var/lib/kubelet/config.yaml。要使用的文件取决于服务器类型。
# For Aerial Devkit servers $ cat <<EOF | sudo tee -a /var/lib/kubelet/config.yaml # Additional Configuration # Feature Gates featureGates: MemoryManager: true # CPU Manager Configuration cpuManagerPolicy: "static" cpuManagerPolicyOptions: full-pcpus-only: "true" reservedSystemCPUs: 0-2,22-23 # Topology Manager Configuration topologyManagerPolicy: "restricted" topologyManagerScope: "container" # Memory Manager Configuration memoryManagerPolicy: "Static" reservedMemory: - numaNode: 0 limits: memory: 100Mi EOF # for Dell R750 servers $ cat <<EOF | sudo tee -a /var/lib/kubelet/config.yaml # Additional Configuration # Feature Gates featureGates: MemoryManager: true # CPU Manager Configuration cpuManagerPolicy: "static" cpuManagerPolicyOptions: full-pcpus-only: "true" reservedSystemCPUs: 0-3 # Topology Manager Configuration topologyManagerPolicy: "restricted" topologyManagerScope: "pod" # Memory Manager Configuration memoryManagerPolicy: "Static" reservedMemory: - numaNode: 0 limits: memory: 50Mi - numaNode: 1 limits: memory: 50Mi EOF
排空每个工作节点。
# Run from k8s master or other server where you have the kube config kubectl drain $nodeName --force --ignore-daemonsets
重启每个工作节点的 kubelet。
# Run on worker node sudo systemctl stop kubelet sudo rm -f /var/lib/kubelet/cpu_manager_state sudo rm -f /var/lib/kubelet/memory_manager_state sudo systemctl start kubelet sudo systemctl status kubelet
确认 kubelet 状态，验证其是否健康。
$ systemctl status kubelet ● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/kubelet.service.d └─10-kubeadm.conf Active: active (running) since Thu 2023-10-12 19:36:05 UTC; 5s ago
取消节点隔离。
# Run from k8s master or other server where you have the kube config kubectl uncordon $nodeName
6. 设置名为 “regcred” 的注册表密钥，以便能够从 $YourPrivateRegistry 容器注册表中拉取。按照 Kubernetes 文档中描述的步骤操作。如果您使用 $YourPrivateRegistry=nvcr.io，请记住从 NGC API 密钥设置门户生成 API 密钥（如果您还没有）。
构建 Aerial 二进制容器#
本节介绍如何为 cuphycontroller_scf、test_mac 和 ru_emulator 应用程序构建 Aerial 二进制容器，以及一些示例场景测试向量。
提取构建脚本。
mkdir -p cuPHY-CP/ docker pull nvcr.io/ea_aerial_sdk/aerial:24-2-cubb docker run --rm -u `id -u` -v .:/staging nvcr.io/ea_aerial_sdk/aerial:24-2-cubb cp -a /opt/nvidia/cuBB/cuPHY-CP/container /staging/cuPHY-CP/
安装依赖项
sudo apt update sudo apt install python3-pip -y pip3 install hpccm
构建二进制容器并推送到您的私有容器仓库。
AERIAL_BUILD_IMAGE=nvcr.io/ea_aerial_sdk/aerial:24-2-cubb AERIAL_RELEASE_REPO=$YourPrivateRepo/ AERIAL_RELEASE_VERSION_TAG=$YourTag ./cuPHY-CP/container/build_binary.sh docker push $YourPrivateRepo/aerial_binary:$YourTag-amd64
使用 Helm Chart 部署二进制容器#
配置 NGC cli 工具 - 按照 https://ngc.nvidia.com/setup/installers/cli 中的步骤操作
登录到 NGC
$ ngc config set
从 NGC 获取 Helm Chart。
ngc registry chart pull ea_aerial_sdk/aerial-l1 ngc registry chart pull ea_aerial_sdk/aerial-ru-emulator
创建特定于您的环境的值覆盖文件。您必须更改以下值
YourRUEmulatorNodeName
YourPrivateRepo
YourTag
<DU 的 FH 端口的 MAC 地址>
YourDUNodeName
$ cat <<EOF | tee override-ru-emulator-binary.yaml # Deployment customization extraSpec: nodeName: "YourRUEmulatorNodeName" image: repository: YourPrivateRepo/ name: aerial_binary pullPolicy: Always # Overrides the image tag whose default is the chart appVersion. tag: "YourTag" peerethaddr: "<MAC Address of DU's FH Port>" # Spacing is critical below extraSetup: | \`# ru-emulator extra setup\` sed -i "s/enable_beam_forming:.*/enable_beam_forming: 1/" ../cuPHY-CP/ru-emulator/config/config_dyn.yaml \`# Configure NIC PCIe Address\` sed -i "s/nic_interface.*/nic_interface: 0000:3b:00.0/" ../cuPHY-CP/ru-emulator/config/config_dyn.yaml EOF $ cat <<EOF | tee override-l1-binary.yaml # Deployment customization extraSpec: nodeName: "YourDUNodeName" image: repository: YourPrivateRepo/ name: aerial_binary pullPolicy: Always # Overrides the image tag whose default is the chart appVersion. tag: "YourTag" enableTestMACContainer: 1 # Spacing is critical below extraSetup: | \`# Aerial L1 extra setup\` \`# Launch pattern related configuration\` sed -i "s/cell_group_num: .*/cell_group_num: 16/" ../cuPHY-CP/cuphycontroller/config/cuphycontroller_dyncore.yaml; sed -i "s/pusch_nMaxPrb: .*/pusch_nMaxPrb: 136/" ../cuPHY-CP/cuphycontroller/config/cuphycontroller_dyncore.yaml; \`# 3GPP conformance\` sed -i "s/pusch_tdi:.*/pusch_tdi: 1/" ../cuPHY-CP/cuphycontroller/config/cuphycontroller_dyncore.yaml; sed -i "s/pusch_cfo:.*/pusch_cfo: 1/" ../cuPHY-CP/cuphycontroller/config/cuphycontroller_dyncore.yaml; sed -i "s/pusch_to:.*/pusch_to: 1/" ../cuPHY-CP/cuphycontroller/config/cuphycontroller_dyncore.yaml; sed -i "s/puxch_polarDcdrListSz:.*/puxch_polarDcdrListSz: 8/" ../cuPHY-CP/cuphycontroller/config/cuphycontroller_dyncore.yaml; \`# Configure NIC PCIe Address\` sed -i "s/ nic:.*/ nic: 0000:cc:00.1/" ../cuPHY-CP/cuphycontroller/config/cuphycontroller_dyncore.yaml; # Spacing is critical below extraTestMACSetup: | \`# testMAC extra setup\` \`#sed -i "s/test_slots: 0/test_slots: 100000/" ../cuPHY-CP/testMAC/testMAC/test_mac_config_dyncore.yaml;\` sed -i "s/schedule_total_time: 0/schedule_total_time: 470000/" ../cuPHY-CP/testMAC/testMAC/test_mac_config_dyncore.yaml; sed -i "s/fapi_delay_bit_mask: 0/fapi_delay_bit_mask: 0xF/" ../cuPHY-CP/testMAC/testMAC/test_mac_config_dyncore.yaml; sed -i "s/builder_thread_enable: 0/builder_thread_enable: 1/" ../cuPHY-CP/testMAC/testMAC/test_mac_config_dyncore.yaml; EOF
部署 Helm Chart。
helm install aerial-ru-emulator-test aerial-ru-emulator-0.20234.0.tgz -f override-ru-emulator-binary.yaml helm install aerial-l1-test aerial-l1-0.20234.0.tgz -f override-l1-binary.yaml
查看每个容器的日志。
# Run in separate windows kubectl logs aerial-l1-test -f kubectl logs aerial-l1-test -c aerial-testmac-ctr -f kubectl logs aerial-ru-emulator-test -f
完成后，删除 Helm Chart 并销毁 pod。
helm uninstall aerial-l1-test helm uninstall aerial-ru-emulator-test
运行原理#
在 pod 部署时，k8s 将专用 CPU 核心动态分配给 Aerial L1 cuphycontroller_scf 容器和 testMAC 容器（如果已部署）。当容器启动时，$cuBB_SDK/cubb_scripts/autoconfig/auto_assign_cores.py 脚本运行，将 k8s 分配的核心映射到各种 Aerial 功能。 auto_assign_cores.py 脚本使用以下模板配置 YAML 文件
$cuBB_SDK/cuPHY-CP/cuphycontroller/config/cuphycontroller_$configL1.yaml -> $cuBB_SDK/cuPHY-CP/cuphycontroller/config/cuphycontroller_dyncore.yaml
$cuBB_SDK/cuPHY-CP/cuphycontroller/config/$l2adapter_filename -> $cuBB_SDK/cuPHY-CP/cuphycontroller/config/l2_adapter_dyncore.yaml
$cuBB_SDK/cuPHY-CP/testMAC/testMAC/$configMAC -> $cuBB_SDK/cuPHY-CP/testMAC/testMAC/test_mac_config_dyncore.yaml
上面使用的变量来自
$cuBB_SDK：容器中定义的环境变量
$configL1：Helm chart aerial-l1/values.yaml (或 override-l1-binary.yaml，如果被覆盖) 变量 ‘configL1’
$l2adapter_filename：YAML 配置参数 ‘l2adapter_filename’，在模板 cuphycontroller 配置 yaml 中定义。
$configMAC：Helm chart aerial-l1/values.yaml (或 override-l1-binary.yaml，如果被覆盖) 变量 ‘configMAC’
aerial-l1-ctr 容器的 auto_assign_cores.py 脚本的示例运行如下
Detected HT Enabled
Detected Multiple NUMA Nodes: [0, 1]. Will use node 1 for scheduling.
OS core affinity: [5, 7, 9, 11, 13, 15, 17, 53, 55, 57, 59, 61, 63, 65]
OS core affinity for numa node 1: [5, 7, 9, 11, 13, 15, 17, 53, 55, 57, 59, 61, 63, 65]
OS isolated cores: [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67,
68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95]
Tentative primary cores: [5, 7, 9, 11, 13, 15, 17]
Cuphycontroller core assignment strategy for HT enabled:
* 1 low priority primary core (shared with dpdk EAL), HT sibling for h2d_copy thread
* {args.workers_ul_count} UL worker primary cores, HT siblings idle
* {args.workers_dl_count} DL worker primary cores, HT siblings idle
* 1 L2A timer thread primary core, HT sibling for L2A msg processing thread
Need 7 physical cores (plus 0 reserved), potential affinity for 7 isolated physical cores
+-------------+----------------------------------------+-------------+----------------------------------------+
| Primary | Primary Core | Sibling | Sibling Core |
| Core Number | Uses | Core Number | Uses |
+-------------+----------------------------------------+-------------+----------------------------------------+
| 5 | low priority threads (inc. DPDK EAL) | 53 | H2D copy |
+-------------+----------------------------------------+-------------+----------------------------------------+
| 7 | UL Worker | 55 | [idle] |
+-------------+----------------------------------------+-------------+----------------------------------------+
| 9 | UL Worker | 57 | [idle] |
+-------------+----------------------------------------+-------------+----------------------------------------+
| 11 | DL Worker | 59 | [idle] |
+-------------+----------------------------------------+-------------+----------------------------------------+
| 13 | DL Worker | 61 | [idle] |
+-------------+----------------------------------------+-------------+----------------------------------------+
| 15 | DL Worker | 63 | [idle] |
+-------------+----------------------------------------+-------------+----------------------------------------+
| 17 | L2A timer | 65 | L2A msg processing |
+-------------+----------------------------------------+-------------+----------------------------------------+
Parsing cuphycontroller configuration template: /opt/nvidia/cuBB/cuPHY-CP/cuphycontroller/config/cuphycontroller_F08_R750.yaml
Writing cuphycontroller configuration: /opt/nvidia/cuBB/cuPHY-CP/cuphycontroller/config/cuphycontroller_dyncore.yaml
Parsing l2adapter configuration template: /opt/nvidia/cuBB/cuPHY-CP/cuphycontroller/config/l2_adapter_config_F08_R750.yaml
Writing l2adapter configuration: /opt/nvidia/cuBB/cuPHY-CP/cuphycontroller/config/l2_adapter_config_dyncore.yaml
aerial-l1-ctr 容器的 auto_assign_cores.py 脚本的示例运行如下
Detected HT Enabled
Detected Multiple NUMA Nodes: [0, 1]. Will use node 1 for scheduling.
OS core affinity: [19, 21, 23, 67, 69, 71]
OS core affinity for numa node 1: [19, 21, 23, 67, 69, 71]
OS isolated cores: [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67,
68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95]
Tentative primary cores: [19, 21, 23]
testMAC core assignment strategy:
* 1 low priority primary core, HT sibling idle
* 1 mac_recv thread primary core, HT sibling idle
* 1 builder thread primary core, HT sibling idle
Need 3 physical cores (plus 0 reserved), potential affinity for 3 isolated physical cores
+-------------+----------------------------------------+-------------+----------------------------------------+
| Primary | Primary Core | Sibling | Sibling Core |
| Core Number | Uses | Core Number | Uses |
+-------------+----------------------------------------+-------------+----------------------------------------+
| 19 | [testmac] low priority threads | 67 | [idle] |
+-------------+----------------------------------------+-------------+----------------------------------------+
| 21 | [testmac] recv | 69 | [idle] |
+-------------+----------------------------------------+-------------+----------------------------------------+
| 23 | [testmac] builder | 71 | [idle] |
+-------------+----------------------------------------+-------------+----------------------------------------+
Parsing testmac configuration template: /opt/nvidia/cuBB/cuPHY-CP/testMAC/testMAC/test_mac_config.yaml
Writing testmac configuration: /opt/nvidia/cuBB/cuPHY-CP/testMAC/testMAC/test_mac_config_dyncore.yaml