W&B Weave

Visit Website

Introduction

Weave is a toolkit for tracing and evaluating AI agents, built by Weights & Biases. The SDK is open source under Apache-2.0, and traces and evaluations are viewed in the W&B web app, either on W&B’s cloud or in a dedicated or self-managed deployment.

Tracing is the entry point. The README explains that agent conversations, turns, LLM calls and tool calls appear in an Agents view, and that weave.init() is enough to trace supported agent SDKs such as the OpenAI Agents SDK, Claude Agent SDK and Google ADK. For your own code, decorating functions with @weave.op records inputs, outputs, latency, token usage and cost for each call and nests them into trees, which makes it easy to see where an agent went wrong.

Evaluation builds on the same objects. You create a dataset of examples, define scorers that check outputs (exact match, regex, JSON validity, LLM-as-a-judge or any Python function), and run weave.Evaluation against a model or agent function. Results are stored and compared in the UI, so a prompt change or model swap can be judged on the same examples before it ships. Built-in scorers cover common needs such as hallucination, summarisation quality, moderation and context relevance for retrieval.

Weave also supports online evaluation: scorers, called monitors or guardrails depending on how they are used, can run on production traces to track quality over time or block bad outputs. Feedback such as thumbs up or down and human annotations attach to calls, and datasets can be built from real traces.

SDKs are available for Python and TypeScript, and integrations cover many providers and frameworks including OpenAI, Anthropic, Google, Mistral, LangChain, LlamaIndex, DSPy and CrewAI. In the quality loop Weave spans release and production: offline evaluations act as a regression gate for prompts and agents, and online scorers watch behaviour once they are live.

The source code is at wandb/weave on GitHub under the Apache-2.0 licence, with about 1,100 stars; the most recent commit was on 10 November 2026.

Features

1. Automatic tracing: Trace supported agent SDKs and LLM providers with one call.
2. @weave.op: Record inputs, outputs, latency, tokens and cost for any function.
3. Agents view: Conversations, turns and tool calls in one place.
4. Evaluations: Run models or agents against datasets with scorers.
5. Built-in scorers: Hallucination, moderation, relevance and more.
6. Custom scorers: Any Python function or LLM judge.
7. Comparisons: Side-by-side evaluation results across versions.
8. Online monitors: Score production traces continuously.
9. Feedback and annotation: Human labels attached to calls.
10. Python and TypeScript: SDKs for both ecosystems.

Pros

1. Ties tracing to evaluation: Production traces become test datasets.
2. Broad integrations: Works with many providers and agent frameworks.
3. Free to use: Open source under the Apache-2.0 licence, so the core tool costs nothing to use.
4. Actively maintained: The repository had commits as recent as 10 November 2026.

Cons

1. Needs the W&B platform: The UI and storage are part of W&B’s service.
2. Paid tiers for teams: Larger usage moves to paid plans.
3. Scorer quality: LLM judges need calibration against human labels.
4. Many concepts: Ops, objects, datasets, scorers and monitors take time to learn.

Usage and Links (Example)

import weave
from openai import OpenAI

weave.init("checkout-assistant")
client = OpenAI()

@weave.op
def answer(question: str) -> str:
    r = client.chat.completions.create(model="gpt-4o-mini",
        messages=[{"role": "user", "content": question}])
    return r.choices[0].message.content

@weave.op
def mentions_refund(expected: str, output: str) -> dict:
    return {"correct": expected.lower() in output.lower()}

dataset = [{"question": "Can I get my money back?", "expected": "refund"}]
evaluation = weave.Evaluation(dataset=dataset, scorers=[mentions_refund])
import asyncio; asyncio.run(evaluation.evaluate(answer))

Practical tips:

1. Start with tracing in development, then promote interesting traces into a dataset.
2. Keep a fixed regression dataset and run it on every prompt or model change.
3. Validate LLM-judge scorers against a sample of human-labelled outputs.

Resources:

• W&B Weave website
• GitHub repository

Pricing

The Weave SDK is open source under Apache-2.0. W&B’s pricing page lists a Free plan at $0 per month and paid plans starting at $60 per month billed monthly, with a 30-day free trial; academic research use is free.

Recommended for?

1. Teams already using Weights & Biases for ML experiments.
2. Agent builders who need traces and evaluations in one tool.
3. Product teams comparing prompts and models on fixed datasets.
4. Consider self-hosted alternatives such as Langfuse or Phoenix if you need everything on your own servers.

Conclusion

Weave connects the two halves of LLM quality work: understanding what an agent did, and measuring whether a change made it better. Tracing, datasets and scorers share one model, so production problems turn into regression tests.

It fits best for teams comfortable with W&B’s platform; pair it with a calibrated set of scorers and a fixed evaluation dataset to get reliable release gates.

On CheckOps, W&B Weave is filed under the Release & CI gates and Production monitoring & observability stages. Related tools listed here include DeepEval, Promptfoo, Ragas, Inspect, OpenAI Evals and Braintrust. When comparing them, weigh the licence, the languages and platforms each supports, whether it is self-hosted or managed, and how cleanly it slots into your existing CI pipeline. Details on this page were checked against the official website, documentation and repository on 11 November 2026.