DOCA 文档 v2.10.0
NVIDIA 文档中心  NVIDIA 网络  BlueField DPU / SuperNIC 和 DOCA  DOCA 文档 v2.10.0  DOCA DPA 通信

在本页中

DOCA DPA 通信

简介

注意

此功能在 Beta 级别提供支持。

DOCA DPA 设备通信库为通信实用程序（如 RDMA）提供接口。它支持使用通信原语在节点之间进行数据传输。用于通信的主要对象称为 RDMA DPA 句柄，它表示两个节点之间的单向通信管道。这些句柄只能由内核访问，并通过在 DPA 数据路径上设置 DOCA RDMA 上下文来创建。有关更多详细信息，请参阅 DOCA RDMA 文档。

为了跟踪通信完成情况，用户可以将 DOCA RDMA 上下文附加到 DPA 完成上下文中。DPA 完成上下文可以选择与 DPA 线程关联，该线程在通信操作完成时触发。或者，用户可以选择手动轮询完成上下文，而无需将其与 DPA 线程关联。

架构

注意

DOCA DPA SDK 未实现任何多线程同步原语。所有 DOCA DPA 对象都不是线程安全的。开发人员必须确保用户程序和内核的设计都能够防止竞争条件。

DOCA Comms 库和头文件

Untitled_Diagram-1737275099316-version-1-modificationdate-1737275107957-api-v2.png

相关的Host端 API

  • 要在 DPA 上创建 DOCA RDMA 上下文，必须将以下 API 用于 DOCA RDMA 上下文

    复制
    已复制！
                
    
            
    doca_error_t doca_ctx_set_datapath_on_dpa(struct doca_ctx *ctx, struct doca_dpa *dpa)

    注意

    当从 BlueField 运行并使用 SF DOCA 设备创建 DOCA RDMA，或者在 DPA 上创建 DOCA RDMA 上下文时，必须将扩展的 DPA 上下文（在同一 SF DOCA 设备上创建）提供给 doca_ctx_set_datapath_on_dpa() API。

  • 要将 DOCA RDMA 上下文附加到 DPA 完成上下文

    复制
    已复制！
                
    
            
    doca_error_t doca_rdma_dpa_completion_attach(struct doca_rdma *rdma, struct doca_dpa_completion *dpa_comp)

  • 要获取 DPA RDMA 句柄和连接 ID

    复制
    已复制！
                
    
            
    doca_error_t doca_rdma_get_dpa_handle(struct doca_rdma *rdma, doca_dpa_dev_rdma_t *dpa_rdma)
doca_error_t doca_rdma_connection_get_id(const struct doca_rdma_connection *rdma_connection, uint32_t *connection_id)

    在线程内核的相关设备 API 中使用输出参数 dpa_handle 和 connection_id。

    注意

    DPA RDMA 不是线程安全的，因此，不得从不同的内核/线程并发使用。

  • 为了支持 DPA 中的多个 DOCA RDMA 连接管理，DOCA 提供了以下 API

    • 要使用主机 API 获取 RDMA 连接 ID

      复制
      已复制！
                  
      
            
      doca_error_t doca_rdma_connection_get_id(const struct doca_rdma_connection *rdma_connection, uint32_t *connection_id)

      这也可以在设备中使用完成 API 检索

      复制
      已复制！
                  
      
            
      uint32_t doca_dpa_dev_get_completion_user_data(doca_dpa_dev_completion_element_t comp_element)

      返回值应用于区分哪个 DOCA RDMA 连接触发了此完成。

    • 要检索 RDMA 完成的 RDMA 工作请求索引

      复制
      已复制！
                  
      
            
      uint32_t doca_dpa_dev_rdma_completion_get_wr_index(doca_dpa_dev_completion_element_t comp_element)

      work request index 应用于获取触发此完成的 DOCA RDMA 上下文的操作索引。

设备端 API

DOCA DPA 为每个设备 RDMA 操作提供两种工作模型

  • 使用 DOCA 缓冲区的 RDMA 操作 API

  • 使用 DOCA mmap 和显式内存地址的 RDMA 操作 API

用户可以使用以下枚举控制 work request submit 配置

复制
已复制！
            

            
/**
 * @brief DPA submit flag type
 */
__dpa_global__ enum doca_dpa_dev_submit_flag {
    DOCA_DPA_DEV_SUBMIT_FLAG_NONE = 0U,
    DOCA_DPA_DEV_SUBMIT_FLAG_FLUSH = (1U << 0),           /**
                                * Use flag to inform related DPA context
                                * (such as RDMA or DPA Async ops) to flush related
                                * operation and previous operations to HW,
                                * otherwise the context may aggregate the operation
                                * and not flush it immediately
                                */
    DOCA_DPA_DEV_SUBMIT_FLAG_OPTIMIZE_REPORTS = (1U << 1), /**
                                * Use flag to inform related DPA context that it may
                                * defer completion of the operation to a later time. If
                                * flag is not provided then a completion will be raised
                                * as soon as the operation is finished, and any
                                * preceding completions that were deferred will also be
                                * raised. Use this flag to optimize the amount of
                                * completion notifications it receives from HW when
                                * submitting a batch of operations, by receiving a
                                * single completion notification on the entire batch.
                                */
};

此枚举可用于以下设备 API 中的 flags 参数

  • 要聚合多个操作并使用 DOCA_DPA_DEV_SUBMIT_FLAG_FLUSH 标志将它们一次性刷新到硬件

    复制
    已复制！
                
    
            
    doca_dpa_dev_rdma_post_write(..., DOCA_DPA_DEV_SUBMIT_FLAG_NONE); // This operation is not flushed to HW as DOCA_DPA_DEV_SUBMIT_FLAG_FLUSH was not used
doca_dpa_dev_rdma_post_write(..., DOCA_DPA_DEV_SUBMIT_FLAG_NONE); // This operation is not flushed to HW as DOCA_DPA_DEV_SUBMIT_FLAG_FLUSH was not used
doca_dpa_dev_rdma_post_write(..., DOCA_DPA_DEV_SUBMIT_FLAG_FLUSH); // This operation is flushed to HW together with all non-flushed previous operations as DOCA_DPA_DEV_SUBMIT_FLAG_FLUSH was used

  • 要使用 DOCA_DPA_DEV_SUBMIT_FLAG_OPTIMIZE_REPORTS 标志将操作的完成推迟到稍后时间

    复制
    已复制！
                
    
            
    doca_dpa_dev_rdma_post_write(..., DOCA_DPA_DEV_SUBMIT_FLAG_OPTIMIZE_REPORTS); // This operation is not flushed to HW as DOCA_DPA_DEV_SUBMIT_FLAG_FLUSH was not used
																			  // and will not trigger a completion as DOCA_DPA_DEV_SUBMIT_FLAG_OPTIMIZE_REPORTS was used
doca_dpa_dev_rdma_post_write(..., DOCA_DPA_DEV_SUBMIT_FLAG_OPTIMIZE_REPORTS); // This operation is not flushed to HW as DOCA_DPA_DEV_SUBMIT_FLAG_FLUSH was not used
																			  // and will not trigger a completion as DOCA_DPA_DEV_SUBMIT_FLAG_OPTIMIZE_REPORTS was used
doca_dpa_dev_rdma_post_write(..., DOCA_DPA_DEV_SUBMIT_FLAG_FLUSH); // This operation is flushed to HW together with all non-flushed previous operations as 
																   // DOCA_DPA_DEV_SUBMIT_FLAG_FLUSH was used.
																   // in addition this operation will trigger a completion once its done as DOCA_DPA_DEV_SUBMIT_FLAG_OPTIMIZE_REPORTS was NOT used

  • 要从远程端缓冲区读取到本地缓冲区

    复制
    已复制！
                
    
            
    void doca_dpa_dev_rdma_post_read(doca_dpa_dev_rdma_t rdma,
								 uint32_t connection_id,
				 				 doca_dpa_dev_mmap_t dst_mmap_handle,
								 uint64_t dst_addr,
								 doca_dpa_dev_mmap_t src_mmap_handle,
								 uint64_t src_addr,
								 size_t length,
								 uint32_t flags)
void doca_dpa_dev_rdma_post_buf_read(doca_dpa_dev_rdma_t rdma,
									 uint32_t connection_id,
								     doca_dpa_dev_buf_t dst_buf_handle,
								     doca_dpa_dev_buf_t src_buf_handle,
								     uint32_t flags)

  • 要将本地内存写入远程端缓冲区

    复制
    已复制！
                
    
            
    void doca_dpa_dev_rdma_post_write(doca_dpa_dev_rdma_t rdma,
								  uint32_t connection_id,
           						  doca_dpa_dev_mmap_t dst_mmap_handle,
								  uint64_t dst_addr,
								  doca_dpa_dev_mmap_t src_mmap_handle,
								  uint64_t src_addr,
								  size_t length,
								  uint32_t flags)
void doca_dpa_dev_rdma_post_buf_write(doca_dpa_dev_rdma_t rdma,
									  uint32_t connection_id,
								      doca_dpa_dev_buf_t dst_buf_handle,
								      doca_dpa_dev_buf_t src_buf_handle,
								      uint32_t flags)

  • 要将本地内存写入远程缓冲区并包含立即数据，该数据可在收到此操作的完成时检索

    复制
    已复制！
                
    
            
    void doca_dpa_dev_rdma_post_write_imm(doca_dpa_dev_rdma_t rdma,
									  uint32_t connection_id,
								      doca_dpa_dev_mmap_t dst_mmap_handle,
								      uint64_t dst_addr,
								      doca_dpa_dev_mmap_t src_mmap_handle,
								      uint64_t src_addr,
								      size_t length,
								      uint32_t immediate,
								      uint32_t flags)
void doca_dpa_dev_rdma_post_buf_write_imm(doca_dpa_dev_rdma_t rdma,
			 							  uint32_t connection_id,
										  doca_dpa_dev_buf_t dst_buf_handle,
										  doca_dpa_dev_buf_t src_buf_handle,
										  uint32_t immediate,
										  uint32_t flags)
// use the following API to retrieve immediate data on completion
uint32_t doca_dpa_dev_get_completion_immediate(doca_dpa_dev_completion_element_t comp_element)

  • 要发送本地内存

    复制
    已复制！
                
    
            
    void doca_dpa_dev_rdma_post_send(doca_dpa_dev_rdma_t rdma,
							 	 uint32_t connection_id,
								 doca_dpa_dev_mmap_t mmap_handle,
								 uint64_t addr,
								 size_t length,
								 uint32_t flags)
void doca_dpa_dev_rdma_post_buf_send(doca_dpa_dev_rdma_t rdma,
								     uint32_t connection_id,
								     doca_dpa_dev_buf_t send_buf_handle,
								     uint32_t flags)

  • 要发送带有立即数据的本地内存，该数据可在收到此操作的完成时检索

    复制
    已复制！
                
    
            
    void doca_dpa_dev_rdma_post_send_imm(doca_dpa_dev_rdma_t rdma,
							 		 uint32_t connection_id,
								     doca_dpa_dev_mmap_t mmap_handle,
								     uint64_t addr,
								     size_t length,
								     uint32_t immediate,
								     uint32_t flags)
void doca_dpa_dev_rdma_post_buf_send_imm(doca_dpa_dev_rdma_t rdma,
									     uint32_t connection_id,
										 doca_dpa_dev_buf_t send_buf_handle,
										 uint32_t immediate,
										 uint32_t flags)
// use the following API to retrieve immediate data on completion
uint32_t doca_dpa_dev_get_completion_immediate(doca_dpa_dev_completion_element_t comp_element)

  • 要处理发布 RDMA 接收操作，请使用以下 API

    • 要发布 RDMA 接收操作

      复制
      已复制！
                  
      
            
      void doca_dpa_dev_rdma_post_receive(doca_dpa_dev_rdma_t rdma, doca_dpa_dev_mmap_t mmap_handle, uint64_t addr, size_t length)
void doca_dpa_dev_rdma_post_buf_receive(doca_dpa_dev_rdma_t rdma, doca_dpa_dev_buf_t receive_buf_handle)

    • 要确认发布接收操作已完成（数据已在关联的数据缓冲区上接收）

      复制
      已复制！
                  
      
            
      void doca_dpa_dev_rdma_receive_ack(doca_dpa_dev_rdma_t rdma, uint32_t num_acked)

      此操作释放 DOCA RDMA 上下文中的 num_acked 条目。

  • 要在远程缓冲区上执行原子加法操作

    复制
    已复制！
                
    
            
    void doca_dpa_dev_rdma_post_atomic_fetch_add(doca_dpa_dev_rdma_t rdma,
							 				 uint32_t connection_id,
										     doca_dpa_dev_mmap_t dst_mmap_handle,
										     uint64_t dst_addr,
										     uint64_t value,
										     uint32_t flags)
void doca_dpa_dev_rdma_post_buf_atomic_fetch_add(doca_dpa_dev_rdma_t rdma,
											     uint32_t connection_id,
												 doca_dpa_dev_buf_t dst_buf_handle,
												 uint64_t value,
												 uint32_t flags)

  • 要发出远程事件信号

    复制
    已复制！
                
    
            
    doca_dpa_dev_rdma_signal_<add|set>(doca_dpa_dev_rdma_t rdma, uint32_t connection_id, doca_dpa_dev_sync_event_remote_t remote_sync_event, uint64_t count)

© 版权所有 2025，NVIDIA。 上次更新时间：2025 年 2 月 12 日。
内容在此