Your agent broke last Tuesday.It'll break the same way next Tuesday.
Silt catches agent failures in production, then turns each one into a test that blocks the next deploy.
You already know your agent hallucinates sometimes.
You find out when a user complains, or when someone scrolls far enough through the traces.
Then you fix it, ship it, and have no idea whether it's actually gone.
The failures you find in production never become the tests that stop them recurring.
The loop
One definition. Production and CI.
A signal is a check you write once. It runs on live traffic and it runs in your pipeline. Same definition, both places.
- 01
Signals
Describe the behaviour you care about in plain English. It starts watching live traffic immediately.
- 02
Issues
Fires get clustered, deduped and ranked, then land in Slack with the trace attached.
- 03
Eval set
One click turns the issue into a test case. Real traffic, no fixtures to invent.
- 04
Ship gate
Run it in CI against your next prompt change. Non-zero exit if the failure came back.
$ silt eval run refund-loop --candidate ./prompts/v12 12 cases · 11 passed · 1 regressed ✗ tool_loop fires on case 7 — clean in v11 exit 1
No sampling
Most tools evaluate a slice of your traffic because checking all of it is expensive. We built the pipeline so you don't have to choose. Failures live in the long tail — that's the part sampling throws away.
Bring your own everything
OpenTelemetry in, your own model keys for evaluation, your own bucket for old traces. Point your existing exporter at us and change a header.
Redacted before it's stored
Names, emails and card numbers are replaced at ingest — with stable placeholders, so traces stay readable and gradeable. Your eval sets are safe to commit to a repo.
Join the waitlist
We're onboarding a small number of teams running agents in production, a few at a time. Tell us what you're building — that's what we use to work out who to let in next.
It's early. Some of this is built, some is being built with the first teams using it. If you'd rather see it working before joining, ask us for a demo.
