Prompt Management

Comparing versions tells you which one won; Prompt Management (a prompt registry, under Governance’s Improve tab) is how self-host closes the rest of the loop without ever touching your agent’s code. LangSmith’s Prompt Hub and Langfuse’s own Prompt Management solve the same “we don’t own your agent” problem the same way: become the prompt’s source of truth, so your agent pulls a version at runtime, and treat “improvement” as propose → you approve → publish a new version - never a direct edit to your deployed code. Self-host does the same thing:

Pull prompts at runtime

prompt = client.evaluations.prompts.get("support-agent-system-prompt")
prompt = client.evaluations.prompts.get(prompt.id)  # by id instead of name works too
# use prompt.text as your own agent's system prompt, however you call your LLM

client.evaluations.run(
    dataset_id="...",
    subject={"kind": "custom_agent", "metadata": {
        "promptName": prompt.name,
        "version": f"{prompt.name}@v{prompt.version}",
    }},
).execute(my_agent_fn)

Create a prompt from the SDK (client.evaluations.prompts.create(name, text)) or the dashboard.

Suggest improvement

From the dashboard: Improve → Prompt Management → a prompt’s row menu → Suggest improvement. It gathers real evidence from two places at once - deliberate Evaluate runs tagged with the prompt’s name (as above), and worst-scoring Online evaluator ratings from real production traffic, since a live trace can carry the exact same metadata.promptName tag a run can:

with client.tracer.trace("support-agent", metadata={"promptName": prompt.name}) as span:
    ...  # your agent's own call, using prompt.text as its system prompt

The evidence panel

Both sources are merged, worst-rated first, and load as soon as the dialog opens: this evidence already exists (a plain data read, no judge call), so you don’t have to generate a rewrite just to see what’s going wrong. Evaluate-run evidence defaults to the current published version only, a v3 prompt’s rewrite shouldn’t get polluted by v1 complaints v2 or v3 may have already fixed, automatically widening to every version if there aren’t at least a few examples on the latest one yet. The evidence panel also runs a second, informational judge pass grouping the same examples into a handful of named recurring failure themes (e.g. “Curt or unempathetic tone”), so you can see the shape of the problem before reading every individual justification.

Generate, review, publish

Each example has a checkbox, filterable by source (production monitoring vs. eval dataset runs); only the checked ones feed Generate suggestion, so a rewrite can be scoped to a specific failure mode instead of always using everything gathered. The result includes the rewritten prompt, a short overall summary, and an itemized what changed list (each entry tagged added, tightened, or removed), plus the judge model actually used. The current and proposed prompts are shown as an editable, line-by-line diff (unified or side-by-side): you can edit the proposed (green) lines directly in the diff before publishing, not just accept it verbatim. Nothing is saved until you click Publish as new version. Your agent’s next client.evaluations.prompts.get(name) call picks up the new version immediately. Tagging your run’s metadata.version as <promptName>@v<N> (shown above) means the Compare versions dialog already tells you whether the published rewrite actually scored better, no separate comparison view needed. Before publishing a proposal, validate it: the candidate and the current version run against a golden dataset’s cases under identical conditions, and the verdict ships with the published version.

The same loop from the SDK

Everything above is scriptable - gather the evidence, generate the proposal, and (once a human has signed off however your process does that) publish:

evidence = client.evaluations.prompts.examples(prompt.id)   # merged, worst-rated first
proposal = client.evaluations.prompts.propose(prompt.id)    # one judge call
print(proposal["revisedText"], proposal["reasoning"])

client.evaluations.prompts.publish_version(
    prompt.id,
    text=proposal["revisedText"],
    reasoning=proposal["reasoning"],
    based_on_version=prompt.version,
)

The dashboard flow stays the human-review path (editable diff, scoped evidence checkboxes); the SDK path exists for automation that carries its own approval step - a nightly job opening a PR with the proposed rewrite, for example.

Prompt Registry

Closing the Loop

Prompt Management

Pull prompts at runtime

Suggest improvement

The evidence panel

Generate, review, publish

The same loop from the SDK

​Pull prompts at runtime

​Suggest improvement

​The evidence panel

​Generate, review, publish

​The same loop from the SDK

Pull prompts at runtime

Suggest improvement

The evidence panel

Generate, review, publish

The same loop from the SDK