What Changed: NVIDIA’s Confidential Computing for AI Inference
When AI moves from experimentation to production, data security becomes a critical concern, especially for enterprises handling sensitive information in healthcare, finance, or government. Traditional encryption protects data at rest and in transit, but during AI inference, data must be decrypted in memory for processing, exposing it to anyone with memory access—malicious insiders, compromised operating systems, or hypervisors. NVIDIA’s Confidential Computing (CC) addresses this gap by providing a hardware-rooted security layer that keeps data encrypted even while in use.
On July 2, 2026, NVIDIA published a blog post detailing how CC is engineered for the era of agentic AI, aiming to scale any model securely without significant performance trade-offs. The post includes benchmark data showing that enabling CC results in minimal overhead—up to 98% of baseline performance—which is a key selling point for product builders who cannot afford to sacrifice speed for security.
How It Works: Hardware Root of Trust and Attestation
NVIDIA CC is built into the hardware of Blackwell GPUs, including the RTX PRO 6000, HGX B200, and HGX B300. The HGX B200 and B300 support confidential computing across up to 8 GPUs with NVLink encryption. At the silicon level, each GPU has a private signing key fused during manufacturing, never exposed to software or the host system. This key forms the foundation of an attestation chain that verifies the integrity of the hardware and software environment before any secrets are deployed.
Remote attestation is performed by the NVIDIA Remote Attestation Service (NRAS), which checks a signed evidence bundle—combining the GPU’s hardware report with CPU TEE measurements (AMD SEV-SNP or Intel TDX)—against a known-good reference integrity manifest (RIM). Once the Confidential VM (CVM) is verified as unmodified, secrets like model decryption keys are released. This attestation handshake is typically a one-time startup event, so it does not add latency to individual inference requests.
Performance Optimizations and Benchmark Results
NVIDIA’s blog highlights two main sources of performance impact in CC mode: secure work submission latency and reduced host-to-device bandwidth. To mitigate these, they collaborated with open-source frameworks like SGLang and FlashInfer to implement optimizations:
- CC-safe autotuner timing: FlashInfer uses the GPU global timer register instead of event timers in CC mode, enabling accurate comparison of kernel candidates.
- Async D2H copy worker: SGLang moves per-step token readback off the scheduler’s critical path, restoring compute/copy overlap that CC can otherwise disrupt.
- Piecewise CUDA graph support: SGLang adds CUDA graph replay for prefill and mixed batches, reducing kernel launch overhead amplified in CC mode.
Benchmarks were conducted on an HGX B300 with Blackwell Ultra, using the Qwen 3.5 397B-A17B model at FP8 precision. The setup used a VM with GPU passthrough, Intel TDX, Ubuntu 25.10 host, and SGLang v0.5.12 with specific PRs. Across concurrency levels from 4 to 256 and input/output sequence lengths of 1024/1024 and 8192/1024, the relative performance impact of enabling CC ranged from -1.0% to -8.1% for throughput and TPOT, with most configurations showing less than 5% degradation. The blog notes that smaller units of work are more affected by secure launch overhead, so batching larger amounts of work per GPU launch can reduce the impact.
Practical Considerations for Product Builders
If you are building a product that relies on AI inference and your customers demand strong data privacy, Confidential Computing is worth evaluating. Here are steps to consider:
- Check hardware and software support: CC requires specific NVIDIA Blackwell GPUs and a compatible software stack, including a CPU TEE (AMD SEV-SNP or Intel TDX). Verify that your cloud provider or data center offers these components.
- Benchmark your workloads: The performance impact varies by workload. Test your models with CC enabled to measure throughput, TTFT, and TPOT against your service-level agreements (SLAs). The blog’s benchmarks provide a reference, but your results may differ.
- Integrate with existing security: Work with your security team to understand how CC fits into your overall defense-in-depth strategy, including encryption, identity management, and compliance requirements.
Remember that the blog post is an overview and does not provide exhaustive technical specifications. For detailed deployment guidance, consult NVIDIA’s official documentation and consider engaging with their solution architects.
Limitations and Trade-offs
While NVIDIA CC offers strong security with minimal performance overhead, there are limitations to keep in mind:
- Hardware dependency: CC is only available on specific Blackwell GPUs, so you may need to upgrade your infrastructure.
- Software ecosystem maturity: The optimizations are being integrated into frameworks like SGLang and FlashInfer, but not all inference frameworks may be fully optimized for CC yet.
- Attestation complexity: Setting up remote attestation with NRAS and managing reference integrity manifests adds operational complexity.
- Performance variability: The benchmark results are specific to the tested configuration (Qwen 3.5, SGLang, Intel TDX). Your mileage may vary, especially with different models, batch sizes, or sequence lengths.
Takeaway: Security as a Product Feature
NVIDIA’s announcement underscores that AI security should be a design consideration from the start, not an afterthought. Hardware-rooted Confidential Computing offers a compelling option for protecting data during inference without forcing a painful trade-off with performance. However, its suitability depends on your specific use case, hardware availability, and performance requirements. The best approach is to clarify your customers’ security needs, evaluate the technical feasibility, and then decide whether to adopt CC. By doing so, you can protect sensitive data while keeping your product responsive and competitive.
Sources
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.