Half your sessions get Niro.
The other half don’t
A controlled experiment on your own repositories: same agent, same work, one arm with Niro’s tools removed. Your numbers decide, not ours.
Half your sessions run without Niro
A comparison only means something if one arm genuinely runs without Niro. In the off arm the tool list is filtered down to one, so the agent works exactly as it did before Niro existed.
On: the full surface. Off: mark_task_complete alone, kept so the session still reports how it went. Everything else disappears from tools/list.
A second check rejects any other tool call while the session is in the off arm, so a session that began before the window cannot leak past its assignment.
Both arms record whether the task succeeded, partially succeeded or failed. That is the signal the two arms are compared on.
How each session is assigned to a group
A session id is hashed with SHA-256 and the parity of the first byte picks the arm. The same id always lands in the same arm, so a reconnect cannot quietly move a session mid-measurement. This runs the real calculation.
What the report shows
The figures below are illustrative. Yours come from your own sessions.
The report computes raw deltas. It does not test statistical significance, and a fortnight of one team’s sessions is a signal rather than a proof. It is your signal though, from your repos and your work, which is worth more than a number from ours.
How a run works
Nothing to instrument. The sessions your team already runs are the experiment.
One to ninety days, named. One active run per account, long enough to catch a normal week rather than a good afternoon.
Every MCP session is assigned an arm once at initialize and stamped for its lifetime. Nothing re-rolls mid-session.
Force-enable puts your next session in the on arm for five minutes, so you can watch the difference rather than wait for the hash to pick it.
Ending a run marks it ended. Nothing is deleted, so every past run stays queryable and comparable.
FAQ
Does the group without Niro slow my team down?
It works exactly as it did before Niro existed. In the off arm the tool list is filtered down to mark_task_complete alone, kept so the session still reports how it went. That is the cost of a real control. If someone needs the tools right now, force-enable puts their next session in the on arm for five minutes.
Are the results statistically significant?
No, and the report does not pretend otherwise. It computes raw deltas on three figures: sessions, median session duration and median tool calls per session. A fortnight of one team's sessions is a signal rather than a proof. But it is your signal, from your repos and your work.
Can a session switch groups?
No. A session id is hashed with SHA-256 and the parity of the first byte fixes the arm at initialize, for that session's lifetime. A second gate rejects any other tool call while a session sits in the off arm, so a reconnect cannot quietly move a session mid-measurement.
What happens to a run when it ends?
Ending a run marks it ended and nothing is deleted, so every past run stays queryable and comparable against the next one. One active run per account, over a window of one to ninety days.
Measure Niro on your own repos
Index the repos your team actually works in, open a window, and read your own numbers in a fortnight.