NVIDIA NIM

NIM (NVIDIA Inference Microservices) serves models behind an OpenAI-compatible /v1/chat/completions API, so the client you patch is the ordinary openai Python client pointed at a NIM endpoint - a NIM container on your own GPUs (http://localhost:8000/v1) or NVIDIA’s hosted API (https://integrate.api.nvidia.com/v1). What this integration adds over patching the OpenAI client is attribution: traces are stamped framework: "nvidia-nim", so NIM traffic gets its own row in Monitor’s Platforms chart and the framework filters instead of blending into openai. Install the integration extra (it installs the openai client package; there is no separate NIM SDK dependency):

pip install "agentx-python[nvidia-nim]"

Usage

Call patch_nim_client() once after creating the client. All subsequent chat.completions.create() calls are traced automatically; no changes to individual API calls are needed. Works with both openai.OpenAI and openai.AsyncOpenAI.

import os

from agentx import AgentX
from agentx.integrations.nvidia_nim import patch_nim_client
import openai

agentx = AgentX.from_env()

nim = openai.OpenAI(
    base_url="http://localhost:8000/v1",  # or https://integrate.api.nvidia.com/v1
    api_key=os.environ.get("NVIDIA_API_KEY", "not-needed-for-local-nim"),
)

patch_nim_client(
    nim,
    tracer=agentx.tracer,
    name="nim-support-agent",
    metadata={"env": "production"},
    session_id="session-xyz-789",
)

response = nim.chat.completions.create(
    model="meta/llama-3.1-8b-instruct",
    messages=[{"role": "user", "content": "How do I cancel my subscription?"}],
)
print(response.choices[0].message.content)

agentx.tracer.flush(timeout=10)

A local NIM container needs no real API key; NVIDIA’s hosted endpoint takes your NVIDIA API key (NGC key).

What gets traced

By default, each chat.completions.create() call produces its own trace.

Field	Source
`input`	`messages` kwarg
`output`	The choices’ message content (or a tool-call description if the reply is a pure tool call)
`latencyMs`	Wall-clock time of the API call - measured after the real response comes back, for both sync and async clients
`model`	`model` kwarg (e.g. `meta/llama-3.1-8b-instruct`)
`inputTokens` / `outputTokens`	`response.usage.prompt_tokens` / `.completion_tokens` - NIM returns the OpenAI-shaped usage block
`framework`	Stamped `nvidia-nim` automatically - the platform label on the framework filter and Monitor’s Platforms chart
`metadata.tools`	The request’s `tools=[...]` definitions, captured for the unregistered-tool listing (Tools & MCPs)
`error`	Exception message if the API call raises

NIM reports no prompt-cache fields, so cacheReadTokens/cacheWriteTokens stay unset.

Streaming

Calls made with stream=True are traced too. The patch returns a transparent proxy over the provider’s stream: iterating it, using it as a context manager, calling close(), and reading its attributes all pass straight through to the real stream, and the trace is assembled from the chunks your code actually consumes.

with nim.chat.completions.create(
    model="meta/llama-3.1-8b-instruct",
    messages=[{"role": "user", "content": "Summarize this ticket."}],
    stream=True,
    stream_options={"include_usage": True},  # asks the endpoint to send token usage on the last chunk
) as stream:
    for chunk in stream:
        if chunk.choices:
            print(chunk.choices[0].delta.content or "", end="")

What a streamed trace carries:

Field	Source
`output`	Text deltas concatenated; tool-call deltas merged by index into a `[tool call] name(arguments)` description when the reply is a pure tool call
`inputTokens` / `outputTokens`	The `usage` block on the final chunk - only present when you pass `stream_options={"include_usage": True}`; without it the counts stay unset rather than reading as 0
`latencyMs`	Measured to the last chunk - the response as your user experienced it, not the moment you closed or dropped the stream
`metadata.streaming` / `metadata.timeToFirstTokenMs`	Marks the call as streamed and records the time to the first chunk, on the trace (standalone call) or the `LLM Call N` child span (inside an active span)

If your code stops reading early (break) or an error interrupts the stream, the trace records what was streamed up to that point, and an error is recorded as the trace’s error. A stream that is dropped without being closed still records what it saw when it is garbage-collected.

Multi-call agentic loops

Wrap a multi-call loop in with tracer.trace(...) to collapse every chat.completions.create() call made inside it into one trace instead of one trace per call:

with agentx.tracer.trace("nim-support-agent", span_kind="agent") as span:
    span.input = question
    # ... call nim.chat.completions.create() as many times as needed ...
    span.output = final_answer

Each call inside the span attaches as a real "LLM Call N" child span carrying the NIM model, tokens, and the nvidia-nim label. A patched call made outside any active span opens its own root trace, stamped with span kind llm - it is a bare model call. The patch sees only the model calls. Tool executions and retrievals around them are plain Python code, so record them manually with tracer.trace_tool_call() and tracer.trace_retrieval() - see the Anthropic integration’s tool-use example for the pattern (identical API, different client).

Cost tracking

NIM is typically self-hosted, so there is no per-token provider price to look up. If you want dollar estimates on NIM traces anyway (an internal chargeback rate, or NVIDIA’s hosted API pricing), add the model id to the pricing catalog - unpriced models show no cost rather than a fake $0.00. A NIM endpoint can also serve as a judge or Playground model: register it as a custom provider model with its base_url in the same catalog.

Other routes to AgentX

patch_nim_client is for the raw client. NIM traffic also arrives through integrations you may already run - pick one layer per LLM call so nothing is traced twice:

LiteLLM: the nvidia_nim/<model> provider is traced by AgentXLiteLLMLogger (labeled litellm).
LangChain: ChatNVIDIA from langchain-nvidia-ai-endpoints is an ordinary chat model, traced by AgentXCallbackHandler (labeled langchain).
OpenTelemetry: anything in the NIM/NeMo ecosystem that exports OTLP lands through the OpenTelemetry endpoint.

`patch_nim_client()` reference

patch_nim_client(
    client: openai.OpenAI | openai.AsyncOpenAI,
    tracer: Tracer,
    name: str = "nim-agent",
    metadata: dict | None = None,
    session_id: str | None = None,
) -> None

Parameter	Description
`client`	The `openai.OpenAI()` or `openai.AsyncOpenAI()` instance pointed at your NIM base URL
`tracer`	`agentx.tracer` from your `AgentX` instance
`name`	Label shown in the AgentX UI for every trace
`metadata`	Static key-value metadata attached to every trace
`session_id`	Links traces from the same conversation thread

Patching is idempotent, and the guard is shared with patch_openai_client: whichever of the two patched a given client first wins, so patch each client with the integration that matches where its base_url actually points.

Call tracer.flush() before your process exits in scripts or one-shot jobs. In long-running servers it is not required: traces drain automatically in the background.

Tracing

Framework Integrations

Usage

What gets traced

Streaming

Multi-call agentic loops

Cost tracking

Other routes to AgentX

`patch_nim_client()` reference

​Usage

​What gets traced

​Streaming

​Multi-call agentic loops

​Cost tracking

​Other routes to AgentX

​patch_nim_client() reference

Usage

What gets traced

Streaming

Multi-call agentic loops

Cost tracking

Other routes to AgentX

`patch_nim_client()` reference