The same model, with and without Niro
42 real developer questions across 3 production codebases. The same model and the same prompts on both sides. The only difference is whether Niro’s tools were available. Ground truth verified by reading the source.
Headline results
The baseline with unlimited retries
We asked that too, so we gave the baseline unlimited retries on every contested query and let it run until it matched or gave up.
| Alone, with retries | With Niro | |
|---|---|---|
| Total cost, 36 contested | $14.16 | $4.72 |
| Total time, 36 contested | 56 min | 19 min |
| Queries still failing | 8 of 19 | 0 of 36 |
| First-pass accuracy | - | F1 0.851 |
It is not a context-length ceiling. It is structural: cross-service dependencies, HTTP call chains and event flows are not in any single file, so no number of retries over files will find them.
Results by task type
On tracing, accuracy is near-identical and only efficiency moves. Publishing that is the point. A benchmark that wins everywhere is a benchmark that was designed to.
“What breaks if I change this?”
| Alone | + Niro | ||
|---|---|---|---|
| Accuracy (F1) | 0.774 | 0.781 | +0.7% |
| Cost | $12.77 | $2.38 | 81% cheaper |
| Tool calls | 409 | 68 | 83% fewer |
The biggest cost win. Cross-service impact cannot be resolved by scanning files, so retries burn tokens on the wrong ones.
“Trace the request path through services”
| Alone | + Niro | ||
|---|---|---|---|
| Accuracy (F1) | 0.937 | 0.965 | +2.8% |
| Cost | $1.03 | $1.05 | about the same |
| Tool calls | 138 | 34 | 75% fewer |
Near-perfect on both sides. The gain here is efficiency, not accuracy: a quarter of the calls to reach the same answer.
“What endpoints, collections or bugs exist?”
| Alone | + Niro | ||
|---|---|---|---|
| Accuracy (F1) | 0.857 | 0.935 | +7.8% |
| Cost | $5.41 | $1.39 | 74% cheaper |
| Tool calls | 367 | 67 | 82% fewer |
The largest accuracy gain. Finding what exists across services needs traversal; file search misses what is not locally visible.
Method
Verified by hand: reading the source and tracing call chains, not asking a model to grade itself.
Single run per query. No cherry-picking a best-of.
The baseline ran iterative retries until convergence or timeout, and every retry’s cost is in the totals.
One model, Claude Opus 4.6, and identical prompts. The only variable is whether Niro’s tools were reachable.
Java and Python microservices; a 200k+ line enterprise monorepo; a TypeScript and Firebase full-stack app.
Monthly cost by team size
The per-query savings, extrapolated to a month of normal use at each team size.
| Size | Alone | With Niro | Saved | |
|---|---|---|---|---|
| Solo dev | 1 dev | $113 | $68 | $45/mo |
| Small team | 5 devs | $567 | $342 | $225/mo |
| Engineering org | 50 devs | $5,670 | $3,420 | $2,250/mo |
| Enterprise | 200 devs | $22,680 | $13,680 | $9,000/mo |
All 42 queries
Every query, both sides, and what happened when the baseline was allowed to retry. Five never matched at any number of attempts.
| Query | Task | F1 alone | F1 + Niro | Cost alone | Cost + Niro | Calls | Calls + Niro | After retries |
|---|---|---|---|---|---|---|---|---|
| Change AstParser.parseFile | Impact | 0.571 | 1.000 | $0.37 | $0.08 | 29 | 4 | never matched |
| Change Neo4jService.executeCypherStatements | Impact | 0.900 | 0.900 | $0.30 | $0.07 | 14 | 2 | 0.870 |
| Rename GraphService.getEntryPoints | Impact | 0.800 | 0.800 | $0.10 | $0.06 | 1 | 2 | - |
| Change LlmClient class (cross-service) | Impact | 0.176 | 0.162 | $0.51 | $0.19 | 70 | 2 | never matched |
| Change POST /api/ts-parser/parse | Impact | 0.400 | 0.444 | $0.20 | $0.12 | 5 | 4 | 0.750 |
| Change RcaService.buildCallerChainFromFqn | Impact | 0.500 | 0.800 | $0.23 | $0.12 | 11 | 2 | 0.833 |
| Change NiroMetricsClient.validateApiKey | Impact | 0.333 | 0.500 | $0.26 | $0.18 | 9 | 5 | 1.000 |
| Callers of GraphController.getEntryPoints | Impact | 0.000 | 0.000 | $0.11 | $0.11 | 2 | 2 | - |
| Change MathService.divide (cross-service) | Impact | 0.889 | 1.000 | $0.18 | $0.10 | 30 | 2 | 0.889 |
| Change MathService.square (unused) | Impact | 0.000 | 0.000 | $0.10 | $0.08 | 6 | 2 | - |
| Change DBService.findPersonByName | Impact | 1.000 | 1.000 | $0.15 | $0.09 | 8 | 2 | - |
| Change PersonRepository.findByAgeBetween | Impact | 0.571 | 0.750 | $0.10 | $0.10 | 4 | 2 | 1.000 |
| Change DatabaseServiceClient.getPersonById | Impact | 1.000 | 1.000 | $0.11 | $0.09 | 4 | 2 | - |
| Change SentimentClient.callSentimentService | Impact | 1.000 | 1.000 | $0.11 | $0.09 | 6 | 2 | - |
| Change POST /api/db/save-person | Impact | 1.000 | 1.000 | $0.18 | $0.09 | 32 | 2 | - |
| Change FeedbackCache class | Impact | 1.000 | 1.000 | $0.12 | $0.09 | 6 | 2 | - |
| Change GET /api/math/add | Impact | 0.750 | 1.000 | $0.19 | $0.09 | 35 | 2 | 1.000 |
| Change POST /analyze (Python → Java) | Impact | 0.857 | 1.000 | $0.20 | $0.10 | 34 | 2 | 1.000 |
| Change POST /orders handler | Impact | 0.889 | 0.545 | $0.24 | $0.06 | 43 | 2 | - |
| Change fetchProducts API helper | Impact | 1.000 | 1.000 | $0.13 | $0.05 | 21 | 2 | - |
| Change orders collection schema (cross-svc) | Impact | 0.545 | 0.833 | $0.17 | $0.15 | 27 | 8 | 0.909 |
| Change API_BASE_URL config | Impact | 1.000 | 1.000 | $0.10 | $0.13 | 5 | 7 | - |
| Change Firestore client setup | Impact | 0.667 | 1.000 | $0.09 | $0.09 | 4 | 4 | 1.000 |
| Rename formatTimestamp utility | Impact | 0.667 | 1.000 | $0.11 | $0.05 | 3 | 2 | never matched |
| Trace POST /api/demo/feedback/analyze | Tracing | 1.000 | 0.971 | $0.19 | $0.29 | 18 | 13 | n/a |
| Trace POST /analyze inside Python service | Tracing | 0.744 | 0.944 | $0.11 | $0.10 | 12 | 2 | n/a |
| Trace GET /future-age-analysis/{id} | Tracing | 0.956 | 0.956 | $0.22 | $0.16 | 16 | 4 | n/a |
| Trace NPE in DeepService.startProcess | Tracing | 1.000 | 1.000 | $0.10 | $0.14 | 5 | 4 | n/a |
| Trace GET /calculate-age-statistics | Tracing | 0.960 | 0.960 | $0.23 | $0.17 | 47 | 5 | n/a |
| Trace POST /person/save to MongoDB | Tracing | 0.960 | 0.960 | $0.18 | $0.19 | 40 | 6 | n/a |
| All REST API endpoints (MD4) | Discovery | 0.950 | 0.950 | $0.17 | $0.06 | 17 | 2 | - |
| External HTTP calls from java-debug-demo | Discovery | 0.947 | 1.000 | $0.15 | $0.05 | 26 | 2 | 1.000 |
| External HTTP calls from Python service | Discovery | 1.000 | 1.000 | $0.17 | $0.04 | 8 | 2 | - |
| Functions reading/writing databases | Discovery | 0.857 | 1.000 | $0.18 | $0.24 | 16 | 10 | 1.000 |
| Known bugs and root causes | Discovery | 1.000 | 0.909 | $0.28 | $0.21 | 51 | 17 | - |
| API gateway routing logic | Discovery | 1.000 | 0.857 | $0.27 | $0.13 | 43 | 4 | - |
| All REST API endpoints (ShopLine) | Discovery | 0.000 | 1.000 | $0.21 | $0.05 | 37 | 2 | never matched |
| Firestore collections & service access | Discovery | 0.667 | 1.000 | $0.24 | $0.12 | 33 | 4 | never matched |
| Scheduled tasks & event-triggered fns | Discovery | 1.000 | 1.000 | $0.32 | $0.11 | 47 | 5 | - |
| Frontend HTTP calls to backend | Discovery | 0.857 | 1.000 | $0.18 | $0.05 | 24 | 2 | 1.000 |
| Data access patterns (direct vs API) | Discovery | 0.600 | 1.000 | $0.18 | $0.20 | 31 | 8 | 1.000 |
| Bugs and code quality issues | Discovery | 0.667 | 0.500 | $0.21 | $0.13 | 34 | 9 | - |
42 queries · 24 impact, 12 discovery, 6 tracing · one run each
FAQ
Can I reproduce this on my own repositories?
That is what the A/B run is for. Index the repos your team actually works in, open a window of one to ninety days, and read your own deltas. Forty-two queries on three codebases is a starting point, not a proof about your code.
Why a single run per query?
Because a best-of-three is a choice about which run to publish. One run each, with ground truth verified by hand (reading the source and tracing call chains rather than asking a model to grade itself), and every retry the baseline spent counted in the totals.
Which model, and were the prompts the same?
Claude Opus 4.6 on both sides, through Claude Code, with identical prompts. The only variable is whether Niro's tools were reachable.
Where does Niro barely win?
Execution tracing. F1 goes 0.937 to 0.965 and the cost is about the same; the whole gain is reaching that answer in a quarter of the tool calls. Publishing that is the point, because a benchmark that wins everywhere is a benchmark that was designed to.
Run the comparison on your repos
The A/B test splits your own sessions: half get Niro, half run without it, and your outcome numbers settle it.