为 DeepStream 开发扩展

本节概述了扩展的开发，重点是开发基于 DeepStream 的组件。通过示例代码片段解释了各种接口和组件的用法。

为了简单起见，组件定义及其方法的实现一起显示在示例代码中。但是，可以根据需要将其拆分到头文件和 .cpp 源文件中。

注意

• 作为 NvDsInterfaceExt 一部分提供的 interfaces.hpp 头文件包含基本的 DeepStream 接口定义。

• 某些具体组件类型（如 I/O，NvDsProbeConnector）的定义包含在作为 NvDsBaseExt 一部分提供的头文件中。

• 组件特定的 Action、Signal、PropertyController 和 Configuration 类型的定义包含在组件所属扩展提供的头文件中。

• 所有 DeepStream 类型都在命名空间 nvidia::deepstream 下定义。如果未使用相同的命名空间，则实现应注意这一点。

扩展和组件工厂注册样板代码

以下提供了关于注册扩展和组件的样板代码

#include "gxf/std/extension_factory_helper.hpp"
#include "sample_runtime_source_manipulator.hpp"
...

GXF_EXT_FACTORY_BEGIN()
GXF_EXT_FACTORY_SET_INFO(0x44a711e485194a68, 0x81e8be7ee4af3ff0,
                         "NvDsSampleExt",
                         "Sample components for demonstrating usage of various "
                         "DeepStream interfaces and components",
                         "NVIDIA", "0.0.1", "Proprietary");
...
GXF_EXT_FACTORY_ADD(
    0x717b2c432f104fe8, 0xb96165e408ece299,
    nvidia::deepstream::NvDsSimpleComponent,

    nvidia::deepstream::INvDsComponent,
    "Description of a simple component");
...
GXF_EXT_FACTORY_END()

• 包含帮助宏的头文件 gxf/std/extension_factory_helper.hpp。包含任何其他包含要注册的组件的定义的头文件。

• GXF_EXT_FACTORY_BEGIN() 和 GXF_EXT_FACTORY_END() 标记包含注册代码的代码块的开始和结束。

• 在工厂块内，调用 GXF_EXT_FACTORY_SET_INFO() 以设置扩展信息，包括扩展的 UUID、扩展名称、描述、作者、版本和许可证。

• 对于要注册为扩展一部分的每个组件，调用 GXF_EXT_FACTORY_ADD() 以添加组件以及组件信息，包括组件的 UUID、组件类型、组件的基本类型和组件的描述。

• UUID 是唯一的 128 位标识符。这在所有扩展和注册的组件中必须是唯一的。在代码中，两个 64 位无符号整数表示 128 位 UUID 的高 64 位和低 64 位。

一个简单的 DeepStream 组件

#include "extensions/nvdsinterface/interfaces.hpp"
namespace nvidia {
namespace deepstream {

class NvDsSimpleComponent : INvDsComponent {
 public:
  // Public methods using which other components can interact with this
  // component via its handle.
  void simpleComponentMethod() {
    //
  }

 private:
  gxf_result_t registerInterface(nvidia::gxf::Registrar *registrar) override {
    nvidia::gxf::Expected<void> result;

    result &= registrar->parameter(
        simple_param_,       // Parameter member variable
        "simple-param-key",  // Parameter name(key). This is used to set
                             // parameter value in a graph
        "Simple Parameter",  // Parameter head line
        "Description of the simple parameter",  // A description of the
                                                // parameter
        100UL,                        // A default value for the parameter
        GXF_PARAMETER_FLAGS_OPTIONAL  // Parameter flags marking it
    );

    result &= registrar->parameter(handle_param_, "handle-parameter",
                                   "Handle Parameter",
                                   "Description of the handle parameter",
                                   std::nullopt, GXF_PARAMETER_FLAGS_OPTIONAL);

    return nvidia::gxf::ToResultCode(result);
  }

  gxf_result_t initialize() override {
    // This method can be used to initialize the component.
    ...

    // Check if parameter is set
    if (simple_param_.try_get() != std::nullopt) {
      uint64_t simple_param_value = simple_param_.try_get().value();
      ...
    }

    return GXF_SUCCESS;  // return GXF_FAILURE in case of any fatal error
  }

  gxf_result_t deinitialize() override {
    // This method can be used to deinitialize the component.
    ...

    return GXF_SUCCESS;  // return GXF_FAILURE in case of any fatal error
  }

  gxf_result_t start() override {
    // Start the component. The underlying DeepStream pipeline and other
    // components are already initialized. It is safe to call methods of other components
    // via their handles.
    ...

    // Check if any component is attached to the parameter
    if (handle_param.try_get() != std::nullopt) {
      SampleOtherComponent *other_comp = handle_param.try_get().value();
      other_comp->otherComponentMethod();
      ...
    }

    return GXF_SUCCESS;  // return GXF_FAILURE in case of any fatal error
  }

  gxf_result_t stop() override {
    // Pipeline has been stopped. All the components would be deinitialized
    // after this. Add any logic for stopping the component.

    return GXF_SUCCESS;  // return GXF_FAILURE in case of any fatal error
  }

  nvidia::gxf::Parameter<uint64_t> simple_param_;
  nvidia::gxf::Parameter<nvidia::gxf::Handle<SampleOtherComponent>>
      handle_param_;
};

}  // namespace deepstream

INvDsInPlaceDataHandler 的实现

#include "extensions/nvdsbase/nvds_probe_connector.hpp"

#include "extensions/nvdsinterface/interfaces.hpp"

namespace nvidia {
namespace deepstream {

class NvDsSampleInPlaceDataHandler : public INvDsInPlaceDataHandler {
  nvidia::gxf::Parameter<nvidia::gxf::Handle<NvDsProbeConnector>>
      probe_connector_;

  gxf_result_t registerInterface(nvidia::gxf::Registrar *registrar) {
    ...
    // Must register a handle parameter of type NvDsProbeConnector
    result &= registrar->parameter(
        probe_connector_, "probe-connector", "Probe Connector",
        "Handle to a nvidia::deepstream::NvDsProbeConnector component",
        std::nullopt, GXF_PARAMETER_FLAGS_OPTIONAL);
    ...
  }

  gxf_result_t initialize() override {
    ...

    // Need to set the handler and flags which tells NvDsProbe component
    // what type of data this component wants to handle. This must be done
    // in the initialize method.
    if (probe_connector_.try_get() != std::nullopt) {
      // The pointer to self (this) can be passed since the component
      // implements the INvDsInPlaceDataHandler interface.
      probe_connector_.try_get().value()->set_handler(this);
      probe_connector_.try_get().value()->set_flags(static_cast<NvDsProbeFlags>(
          NvDsProbeFlags::BUFFER | NvDsProbeFlags::EVENT));
    }
    ...
    return GXF_SUCCESS;
  }

  bool handle_buffer(GstPad *pad, nvidia::gxf::Entity buffer_data) override {
    // Check for presence of NvDsBatchMeta and NvBufSurface in the buffer_data
    // entity since this implementation requires it. Other implementations may
    // want to check for presence of other data components.
    if (!buffer_data.get<NvDsBatchMetaHandle>(BUF_DATA_KEY_VIDEO_BATCH_META)) {
      return true;
    }
    if (!buffer_data.get<NvBufSurfaceHandle>(BUF_DATA_KEY_NVBUFSURFACE)) {
      return true;
    }
    NvDsBatchMeta *batch_meta =
        *(buffer_data.get<NvDsBatchMetaHandle>(BUF_DATA_KEY_VIDEO_BATCH_META)
              .value());
    NvBufSurface *surf =
        *(buffer_data.get<NvBufSurfaceHandle>(BUF_DATA_KEY_VIDEO_BATCH_META)
              .value());

    // Use batch_meta and surf
    ...
    return true; // allows buffer to pass through. return false to drop the buffer
  }

  bool handle_event(GstPad *pad, GstEvent *event) override {
    // Use the event
    ...
    return true; // allows buffer to pass through. return false to drop the event
  }
};

}  // namespace deepstream
}  // namespace nvidia

作为示例扩展一部分提供的组件 NvDsSampleProbeMessageMetaCreation 是 INvDsInPlaceDataHandler 接口的完整示例实现。

控制属性

此示例代码演示了自定义组件如何使用 INvDsPropertyController（在本例中为 NvDsOSDPropertyController）组件来控制基于 INvDsElement 的组件（在本例中为 NvDsOSD）的属性。

// Provided as part of the NvDsVisualizationExt extension which contains the
// NvDsOSD component. This header contains the definition of
// NvDsOSDPropertyController component which provides methods to control
// properties of NvDsOSD component.
#include "nvdsvisualization_prop_controllers.hpp"

namespace nvidia {
namespace deepstream {

class NvDsSamplePropertyControl : public INvDsComponent {
  nvidia::gxf::Parameter<nvidia::gxf::Handle<NvDsOSDPropertyController>>
      osd_property_controller_;
  NvDsOSDPropertyController *osd_property_controller = nullptr;

  gxf_result_t registerInterface(nvidia::gxf::Registrar *registrar) override {
    ...
    // Must register a handle parameter of type NvDsOSDPropertyController
    result &= registrar->parameter(
        osd_property_controller_, "nvdsosd-prop-controller",
        "NvDsOSD Property Controller",
        "Handle to a nvidia::deepstream::NvDsOSDPropertyController "
        "component.",
        std::nullopt, GXF_PARAMETER_FLAGS_OPTIONAL);
    ...
  };

  void sample_thread_func() {
    while (1) {
      bool display_text;
      // Get the current property value
      osd_property_controller->get_display_text(&display_text);
      // Toggle the property value
      osd_property_controller->set_display_text(!display_text);

      sleep(1);
      // Break when signalled to stop
    }
  }

  gxf_result_t start() override {
    ...
    // Check if the property controller component is attached
    if (osd_property_controller_.try_get() != std::nullopt) {
      osd_property_controller = osd_property_controller_.try_get().value();
      osd_property_controller->set_display_text(true);

      // Start a thread which periodically toggles the display-text property
    }
    ...
  }

  gxf_result_t stop() override {
    // Signal samplethread to stop
  }
};

}  // namespace deepstream
}  // namespace nvidia

触发操作

此示例代码演示了自定义组件如何使用 INvDsAction（在本例中为 NvDsSourceManipulationAction）组件来触发基于 INvDsElement 的组件（在本例中为 NvDsMultiSrcInput）的操作。

   // Provided as part of the NvDsSourceExt extension which contains the
   // NvDsMultiSrcInput component. This header contains the definition of
  // NvDsSourceManipulationAction component which provides methods to trigger
  // add/remove source actions of NvDsMultiSrcInput component.
  #include "nvdsinputsrc_signals.hpp"

  namespace nvidia {
  namespace deepstream {

  class NvDsSampleSourceManipulator : public INvDsComponent {

  nvidia::gxf::Parameter<nvidia::gxf::Handle<NvDsSourceManipulationAction>>
source_manip_action_;
  NvDsSourceManipulationAction *source_manip_action = nullptr;

  bool add = true;

  gxf_result_t registerInterface(nvidia::gxf::Registrar *registrar) override {
  ...
  // Must register a handle parameter of type NvDsSourceManipulationAction
      result &= registrar->parameter(
  source_manip_action_, "source-manip-action", "Source Manipulation Action",
  "Handle to a nvidia::deepstream::NvDsSourceManipulationAction "
"component",
std::nullopt, GXF_PARAMETER_FLAGS_OPTIONAL);
  ...
  }

  void sample_thread_func() {
          while (1) {
      // Call methods of the action component via it's handle to trigger the add/remove actions.
      if (add) {
      if (!source_manip_action->add_source("file:///some/file/path.mp4", 2)) {
    GXF_LOG_WARNING("Failed to add source");
      }
      } else {
      if (!source_manip_action->remove_source("file:///some/file/path.mp4", 2)) {
      GXF_LOG_WARNING("Failed to remove source");
      }
  }
  add = !add;

  sleep(10);
  // Break when signalled to stop
  }
  }

  gxf_result_t start() override {
  ...
  // Check if the action component is attached
  if (source_manip_action_.try_get() != std::nullopt) {
   source_manip_action = source_manip_action_.try_get().value();
  // Start a thread to periodically add / remove a source
  }
  ...
  }

  gxf_result_t stop() override {
  // Stop the thread
  }
  };

  }  // namespace deepstream
  }  // namespace nvidia

作为示例扩展一部分提供的组件 NvDsSampleSourceManipulator 是演示此功能的完整示例实现。

处理信号回调

此示例代码演示了自定义组件如何使用 INvDsSignal（在本例中为 NvDsModelUpdatedSignal）组件来处理基于 INvDsElement 的组件（在本例中为 NvDsInferVideo）发出的信号。

// Provided as part of the NvDsInferenceExt extension which contains the
// NvDsInferVideo component. This header contains the definition of
// NvDsModelUpdatedSignal component which provides the Handler interface and
// callback method prototype and a way to set the Handler instance.
#include "nvdsinference_signals.hpp"

namespace nvidia {
namespace deepstream {

class NvDsSampleSignalHandler : public INvDsComponent,
                       public NvDsModelUpdatedSignal::Handler {
  nvidia::gxf::Parameter<nvidia::gxf::Handle<NvDsModelUpdatedSignal>> model_update_signal_;

  gxf_result_t registerInterface(nvidia::gxf::Registrar *registrar) override {
    ...
    // Must register a handle parameter of type NvDsModelUpdatedSignal
    result &= registrar->parameter(
        model_update_signal_, "model-update-signal",
        "Model Updated Signal",
        "Handle to a nvidia::deepstream::NvDsModelUpdatedSignal "
        "component.",
        std::nullopt, GXF_PARAMETER_FLAGS_OPTIONAL);
    ...
  };

  // Implement the methods of the NvDsModelUpdatedSignal::Handler interface
  void on_model_updated(int errorCode, gchar *configFilePath) override {
    // Handle the on-the-fly model update status.
  }

  gxf_result_t start() override {
    // Start the component
    ...

    // Check if the signal component is attached, set the signal handler to self
    if (model_update_signal_.try_get() != std::nullopt) {
      // The pointer to self (this) can be passed since the component
      // implements the NvDsModelUpdatedSignal::Handler interface
      model_update_signal_.try_get().value()->set_handler(this);
    }
    ...
  }

  gxf_result_t stop() override {
    // Stop the component
  }
};

}  // namespace deepstream
}  // namespace nvidia

配置提供程序组件的实现

此示例代码演示了自定义组件如何实现配置提供程序接口（在本例中为 INvDsInferModelConfigComponent）组件，以提供配置（在本例中为 NvDsInferVideo）。

// Provided as part of the NvDsInferenceExt extension which contains the
// NvDsInferVideo component. This header contains the definition of
// INvDsInferModelConfigComponent interface which acts as a model configuration
// provider to the NvDsInferVideo component.
  #include "nvdsinference_config.hpp"

  namespace nvidia {
  namespace deepstream {

  class SampleModel : public INvDsInferModelConfigComponent {
    gxf_result_t fill_config(NvDsInferModelConfig *config) override {
      // config_infer_primary.txt is packaged alongside the extension binary,
      // along with caffemodel, prototxt and labels file. Use get_absolute_path()
      // method to convert a path relative to the extension binary to an absolute
      // path at runtime.
      config->config_file_path = get_absolute_path("config_infer_primary.txt");
      if (engine_file_.try_get() != std::nullopt) {
        // The configuration provider component itself can have parameters which
        // can be used to populate the configuration.
        config->model_engine_path =
            get_absolute_path(engine_file_.try_get().value());
      }
      return GXF_SUCCESS;
    }

    gxf_result_t registerInterface(nvidia::gxf::Registrar *registrar) override {
      nvidia::gxf::Expected<void> result;

      result &= registrar->parameter(engine_file_, "model-engine-file",
                         "Model Engine File",
                         "Path to the model engine file. Absolute or "
                         "relative to the extension directoy.",
                         std::nullopt, GXF_PARAMETER_FLAGS_OPTIONAL);

      return nvidia::gxf::ToResultCode(result);
    }

    nvidia::gxf::Parameter<std::string> engine_file_;
  };

  }  // namespace deepstream
  }  // namespace nvidia

NvDsSampleAudioTemplateLib 和 NvDsSampleVideoTemplateLib 是作为配置提供程序的其他组件示例。这些组件是示例扩展的一部分。