NVIDIA has released a collection of open-source C++ samples, called Do Inference Now (DIN) Deploy, to streamline the creation of local artificial intelligence applications that run with hardware-accelerated performance on NVIDIA GPUs. According to an NVIDIA technical blog, these samples provide a clear workflow for developers on Windows and Linux to take AI models from popular repositories like Hugging Face and embed them into native C++ applications. The architecture achieves this by combining the cross-platform ONNX Runtime with NVIDIA's specialized TensorRT RTX execution provider, establishing a two-stage process that separates Python-based model preparation from high-performance C++ inference deployment.

The Architectural Philosophy: Separating Model Prep from Deployment

The design of the DIN Deploy samples is intentionally divided into two distinct phases: a model export stage handled in Python and an inference deployment stage built in C++. This separation of concerns is a core principle that offers significant practical advantages for developers. It allows teams to leverage the vast and flexible Python ecosystem, including popular libraries and model hubs like Hugging Face, for the complex task of preparing and converting AI models. Once a model is prepared, it is handed off as a standardized artifact to be integrated into a high-performance, native C++ application designed for efficient execution.

This architectural choice reflects the broader philosophy of the NVIDIA TensorRT ecosystem itself. Official NVIDIA documentation explains that the TensorRT workflow is fundamentally broken into two parts: the various methods for converting a trained model into an optimized engine, and the different runtimes that can be used to deploy that engine. The DIN Deploy samples provide a concrete and accessible implementation of this concept. Furthermore, this approach helps isolate vendor-specific code. A summary of the project notes that this design keeps specialized CUDA code confined to optional accelerated paths, allowing the main C++ application to be built on more general APIs for greater portability.

DIN Deploy Workflow: Model Export vs. Inference Deployment

To understand how the DIN Deploy samples function, it is essential to view the process as two separate but connected stages. The first stage uses Python to prepare the AI model, transforming it from its original format into a portable standard. The second stage takes this standardized model and runs it within a native C++ application, leveraging NVIDIA hardware for accelerated performance. This table breaks down the responsibilities and components of each phase.

A breakdown of the two primary stages in the NVIDIA DIN Deploy workflow, detailing the purpose, components, and output of each phase.
Stage Purpose Key Components Output
Model Export (Python) To convert pre-trained model checkpoints from sources like Hugging Face into a standardized, portable format for inference. A Python exporter script and Hugging Face model checkpoints. An optimized ONNX model artifact ready for deployment.
Inference Deployment (C++) To execute the ONNX model with hardware acceleration inside a native application on Windows or Linux systems. A C++ command-line interface, ONNX Runtime, and the NVIDIA TensorRT RTX execution provider. A local, hardware-accelerated AI application performing inference.

Stage 1: Preparing Models with the Python Exporter

The journey from a conceptual AI model to a deployable asset begins in the Python environment. Each DIN Deploy sample includes a Python exporter script designed to handle the initial conversion process. According to NVIDIA's announcement, this script takes a model checkpoint, such as one from the Hugging Face hub, and transforms it into an ONNX artifact. ONNX, or the Open Neural Network Exchange, is an open standard for representing machine learning models. Its role here is critical, as it creates a portable and interoperable file that decouples the model from its original training framework.

This stage effectively acts as a bridge between the world of machine learning research, which is predominantly Python-based, and the world of native application development. The Python exporter manages the complexities of downloading the model and converting its architecture and weights into the standardized ONNX format. This process ensures that the C++ application in the next stage does not need to understand the specifics of different model training frameworks, but only how to interact with the standardized ONNX file.

Stage 2: C++ Inference with ONNX Runtime and TensorRT RTX

Once the ONNX model artifact is created, the workflow transitions to C++. The DIN Deploy samples provide a native C++ command-line interface (CLI) that serves as the foundation for a local AI application. This C++ code is responsible for loading the ONNX model and executing inference. According to NVIDIA, the application is built upon the ONNX Runtime's session and tensor APIs. ONNX Runtime is a high-performance inference engine that can execute models in the ONNX format across a wide variety of hardware and operating systems, including Windows and Linux, with one report noting support for Arm64 as well.

While ONNX Runtime provides the framework for inference, the actual hardware acceleration on NVIDIA GPUs is unlocked by a specific component: the NVIDIA TensorRT RTX execution provider. This provider is a plug-in for ONNX Runtime that delegates the heavy computational work to NVIDIA's TensorRT, a software development kit for high-performance deep learning inference. By using this execution provider, the C++ application can achieve significant performance gains without requiring developers to write complex, low-level CUDA code directly in their main application logic. The result is a native, hardware-accelerated AI application capable of running efficiently on a user's local machine.

Implementing Local AI with DIN Deploy Samples

Developers can leverage NVIDIA's DIN Deploy samples to build local C++ AI applications by following a two-stage workflow: Python-based model export and C++ inference deployment. The architectural separation allows developers to leverage Python's ecosystem for model handling while benefiting from C++ performance and NVIDIA GPU acceleration for deployment. This approach provides a structured path for creating powerful, native AI features that run directly on a user's hardware, offering benefits in latency, privacy, and offline capability.

The ultimate goal of this workflow is the creation of an efficient, self-contained application. The measurable indicator of success is the successful execution of a DIN Deploy sample application on an NVIDIA GPU, demonstrating accelerated inference for supported AI tasks. By providing these open-source examples, NVIDIA offers a practical and powerful starting point for developers looking to integrate advanced AI directly into their C++ projects for Windows and Linux.

Sources