> ## Documentation Index
> Fetch the complete documentation index at: https://developers.agentx.so/llms.txt
> Use this file to discover all available pages before exploring further.

# NVIDIA NIM

> Auto-trace NVIDIA NIM inference microservices with patch_nim_client

NIM (NVIDIA Inference Microservices) serves models behind an OpenAI-compatible `/v1/chat/completions` API, so the client you patch is the ordinary `openai` Python client pointed at a NIM endpoint - a NIM container on your own GPUs (`http://localhost:8000/v1`) or NVIDIA's hosted API (`https://integrate.api.nvidia.com/v1`). What this integration adds over [patching the OpenAI client](/sdk/integrations/openai) is attribution: traces are stamped `framework: "nvidia-nim"`, so NIM traffic gets its own row in Monitor's Platforms chart and the framework filters instead of blending into `openai`.

Install the integration extra (it installs the `openai` client package; there is no separate NIM SDK dependency):

```bash theme={null}
pip install "agentx-python[nvidia-nim]"
```

## Usage

Call `patch_nim_client()` once after creating the client. All subsequent `chat.completions.create()` calls are traced automatically; no changes to individual API calls are needed.

Works with both `openai.OpenAI` and `openai.AsyncOpenAI`.

```python theme={null}
import os

from agentx import AgentX
from agentx.integrations.nvidia_nim import patch_nim_client
import openai

agentx = AgentX.from_env()

nim = openai.OpenAI(
    base_url="http://localhost:8000/v1",  # or https://integrate.api.nvidia.com/v1
    api_key=os.environ.get("NVIDIA_API_KEY", "not-needed-for-local-nim"),
)

patch_nim_client(
    nim,
    tracer=agentx.tracer,
    name="nim-support-agent",
    metadata={"env": "production"},
    session_id="session-xyz-789",
)

response = nim.chat.completions.create(
    model="meta/llama-3.1-8b-instruct",
    messages=[{"role": "user", "content": "How do I cancel my subscription?"}],
)
print(response.choices[0].message.content)

agentx.tracer.flush(timeout=10)
```

A local NIM container needs no real API key; NVIDIA's hosted endpoint takes your NVIDIA API key (NGC key).

## What gets traced

By default, each `chat.completions.create()` call produces its own trace.

| Field                          | Source                                                                                                                                     |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `input`                        | `messages` kwarg                                                                                                                           |
| `output`                       | The choices' message content (or a tool-call description if the reply is a pure tool call)                                                 |
| `latencyMs`                    | Wall-clock time of the API call - measured after the real response comes back, for both sync and async clients                             |
| `model`                        | `model` kwarg (e.g. `meta/llama-3.1-8b-instruct`)                                                                                          |
| `inputTokens` / `outputTokens` | `response.usage.prompt_tokens` / `.completion_tokens` - NIM returns the OpenAI-shaped usage block                                          |
| `framework`                    | Stamped `nvidia-nim` automatically - the [platform label](/trace/platform-detection) on the framework filter and Monitor's Platforms chart |
| `metadata.tools`               | The request's `tools=[...]` definitions, captured for the unregistered-tool listing (Tools & MCPs)                                         |
| `error`                        | Exception message if the API call raises                                                                                                   |

NIM reports no prompt-cache fields, so `cacheReadTokens`/`cacheWriteTokens` stay unset.

## Streaming

Calls made with `stream=True` are traced too. The patch returns a transparent proxy over the provider's stream: iterating it, using it as a context manager, calling `close()`, and reading its attributes all pass straight through to the real stream, and the trace is assembled from the chunks your code actually consumes.

```python theme={null}
with nim.chat.completions.create(
    model="meta/llama-3.1-8b-instruct",
    messages=[{"role": "user", "content": "Summarize this ticket."}],
    stream=True,
    stream_options={"include_usage": True},  # asks the endpoint to send token usage on the last chunk
) as stream:
    for chunk in stream:
        if chunk.choices:
            print(chunk.choices[0].delta.content or "", end="")
```

What a streamed trace carries:

| Field                                                | Source                                                                                                                                                                |
| ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `output`                                             | Text deltas concatenated; tool-call deltas merged by index into a `[tool call] name(arguments)` description when the reply is a pure tool call                        |
| `inputTokens` / `outputTokens`                       | The `usage` block on the final chunk - only present when you pass `stream_options={"include_usage": True}`; without it the counts stay unset rather than reading as 0 |
| `latencyMs`                                          | Measured to the **last chunk** - the response as your user experienced it, not the moment you closed or dropped the stream                                            |
| `metadata.streaming` / `metadata.timeToFirstTokenMs` | Marks the call as streamed and records the time to the first chunk, on the trace (standalone call) or the `LLM Call N` child span (inside an active span)             |

If your code stops reading early (`break`) or an error interrupts the stream, the trace records what was streamed up to that point, and an error is recorded as the trace's `error`. A stream that is dropped without being closed still records what it saw when it is garbage-collected.

## Multi-call agentic loops

Wrap a multi-call loop in `with tracer.trace(...)` to collapse every `chat.completions.create()` call made inside it into **one** trace instead of one trace per call:

```python theme={null}
with agentx.tracer.trace("nim-support-agent", span_kind="agent") as span:
    span.input = question
    # ... call nim.chat.completions.create() as many times as needed ...
    span.output = final_answer
```

Each call inside the span attaches as a real `"LLM Call N"` child span carrying the NIM model, tokens, and the `nvidia-nim` label. A patched call made outside any active span opens its own root trace, stamped with [span kind](/trace/span-kinds) `llm` - it is a bare model call.

The patch sees only the model calls. Tool executions and retrievals around them are plain Python code, so record them manually with `tracer.trace_tool_call()` and `tracer.trace_retrieval()` - see the [Anthropic integration's tool-use example](/sdk/integrations/anthropic#tool-use) for the pattern (identical API, different client).

## Cost tracking

NIM is typically self-hosted, so there is no per-token provider price to look up. If you want dollar estimates on NIM traces anyway (an internal chargeback rate, or NVIDIA's hosted API pricing), add the model id to the [pricing catalog](/evaluation/model-portability#the-pricing-catalog) - unpriced models show no cost rather than a fake \$0.00. A NIM endpoint can also serve as a judge or Playground model: register it as a `custom` provider model with its `base_url` in the same catalog.

## Other routes to AgentX

`patch_nim_client` is for the raw client. NIM traffic also arrives through integrations you may already run - pick **one** layer per LLM call so nothing is traced twice:

* **LiteLLM**: the `nvidia_nim/<model>` provider is traced by [`AgentXLiteLLMLogger`](/sdk/integrations/litellm) (labeled `litellm`).
* **LangChain**: `ChatNVIDIA` from `langchain-nvidia-ai-endpoints` is an ordinary chat model, traced by [`AgentXCallbackHandler`](/sdk/integrations/langchain) (labeled `langchain`).
* **OpenTelemetry**: anything in the NIM/NeMo ecosystem that exports OTLP lands through the [OpenTelemetry endpoint](/trace/opentelemetry).

## `patch_nim_client()` reference

```python theme={null}
patch_nim_client(
    client: openai.OpenAI | openai.AsyncOpenAI,
    tracer: Tracer,
    name: str = "nim-agent",
    metadata: dict | None = None,
    session_id: str | None = None,
) -> None
```

| Parameter    | Description                                                                           |
| ------------ | ------------------------------------------------------------------------------------- |
| `client`     | The `openai.OpenAI()` or `openai.AsyncOpenAI()` instance pointed at your NIM base URL |
| `tracer`     | `agentx.tracer` from your `AgentX` instance                                           |
| `name`       | Label shown in the AgentX UI for every trace                                          |
| `metadata`   | Static key-value metadata attached to every trace                                     |
| `session_id` | Links traces from the same conversation thread                                        |

Patching is idempotent, and the guard is shared with `patch_openai_client`: whichever of the two patched a given client first wins, so patch each client with the integration that matches where its `base_url` actually points.

<Note>
  Call `tracer.flush()` before your process exits in scripts or one-shot jobs.
  In long-running servers it is not required: traces drain automatically in the
  background.
</Note>