Drop in the trace file. GetEvals works out how your agent actually fails, turns those failures into evals, and hands back a quality report scorecard.
JSONL · JSON · NDJSON · CSV · TSV · TXT · GZ · ZIP
A report being built. Yours takes about ten minutes.
02 — What you get
Scores, ranked failure modes, a heatmap you can click into, and the evals to keep testing with.
Scoreboard — the answer in four numbers
Failure modes — ranked by how often they happen
Heatmap — which trace hit which mode, click any cell
Every verdict — with the line it rests on
assistant: “Your order is still processing — it should ship in two to three days.”
↳ get_order_status returned state: "delivered" four turns earlier
Example report. Yours is built from your own traces.
03 — The run
04 — Trust
05 — Lab
Three runs against public research corpora and one we built ourselves, each reproducible with a single command.
The Lab — every experiment, and what building this taught us →
06 — Questions
OpenAI chat and Batch JSONL, OpenAI Evals files, Anthropic Messages, Claude Code and Codex sessions, Langfuse and LangSmith exports, OpenTelemetry GenAI, OpenInference and Vercel AI SDK spans, ShareGPT and ChatML, and plain CSV — twelve formats in all. Gzip and zip are unpacked for you, and the format is detected automatically.
Thirty shows real patterns; a hundred or two makes the percentages worth quoting. Files up to 200MB are fine — above 200 traces a seeded sample is analysed and the report lists exactly which traces went in.
Nowhere. It is parsed in your browser and never uploaded whole. To grade a trace, a bounded excerpt of that one trace goes to the judging model through our proxy, which stores none of it. No account, no database, no training on your data — close the tab and it is gone. If your traces cannot leave your network at all, redact them first or run the whole thing yourself: it is one Next.js app and your own model key.
Nothing, and there is no account. The model spend is ours, so a run is capped at 120 traces read, 800 cells graded and 5 runs a day. Need more? Clone the repo and use your own key.
Not on its word — on the quote under each verdict, taken from your own trace and checked against your file verbatim. Verdicts are re-judged with the prompt reworded, unstable modes are badged, failed calls are reported as ungraded, and any labels in your upload are used to score the judge. What happened when we did this to ourselves is in the Lab, wins and losses.
No. Those store your traces and watch them over time. This is the one-off analysis that tells you what is worth watching — and hands you the graders to run there.
Yes. Evals data files are read directly, and any label, correct, passed or score field becomes ground truth, so the report can tell you how well the judge matches labels you already trust. Exports come back in the same format, plus promptfoo and portable JSON.
