Opinion: Commitment ledger cuts duplicate work but won’t make coding agents reliable

TL;DR

When multiple AI coding agents talk to each other, they can still duplicate work, get the task order wrong, and waste time. Text chat alone isn't enough to keep them on the same page.

To solve this, I built the ledger around a simple idea. If an agent says, "I'll build the items file," I record that as a commitment instead of leaving it buried in the chat. The implementation is just Python's standard library. No database, API, or external package is involved.

In my test case, using the ledger avoided duplicate work and dependency-notification failures. It didn't fix cases where an agent failed to finish a job or reported work without verification.

That was the key takeaway: a ledger can improve the coordination state, but it can't force an AI agent to follow through on its work.

Why Coordination Fails Even When Communication Doesn't

A team from Stanford and SAP Labs recently published CooperBench, a benchmark for testing whether coding agents can work as teammates rather than as separate solo workers [1]. In the setup, two agents work on the same codebase, each gets a feature, the features can conflict, and the agents can communicate while they work.

The result caught my attention. The cooperating agents did not consistently perform better. In the benchmark, cooperating pairs scored worse than a single agent handling the same total workload [1].

Communication analysis is also important. The paper reports vague or badly timed messages, incorrect assumptions about what another agent is doing, and cases where agents drift away from commitments they had already made [1].

That last part is what led me to this project.

I don't mean that an agent "forgets" in the human sense. The problem is simpler. An agent can say, "I'll implement repository/items.py," and a few messages later there may be nothing outside the conversation that records that decision. Another agent can make the same claim. A dependency can be missed. A completed task can be reported without any separate record showing whether it was actually verified.

The paper also found cases where agents did coordinate successfully. In a minority of runs, agents divided roles, split resources, or negotiated scope without being explicitly instructed to do so [1]. So coordination is possible. The problem is that it isn't reliable.

That gave me a much narrower question to test:

What happens if I take an agent's commitments out of the conversation and give them somewhere to live?

That's what I built with the Commitment Ledger. It is a small deterministic system that records multi-agent coding commitments as explicit state and checks that state for conflicts, dependencies, missed work, and rework.

Full code, tests, and logs are below.

What I Wanted the Ledger to Solve

I wasn't trying to solve every multi-agent failure. I wanted to isolate one specific problem: a commitment exists in the conversation, but another agent has no persistent state to check before taking the same work.

I've seen how this can happen when multiple AI agents work against the same codebase, even when the agents are running in separate terminal sessions and coordinating through a shared document. The failure doesn't announce itself as a coordination bug. It shows up as, "Wait, didn't you already do this?" three turns after the duplicate work happened. Or you discover that a merge has quietly overwritten something another agent finished an hour earlier. By the time you notice it, the wasted work has already happened.

This isn't for a single agent. There is no coordination state to track. It also isn't needed when agents already have strictly partitioned, non-overlapping file ownership, because that solves the conflict problem before it starts.

And it won't tell me whether the work is correct. The results below make that limitation pretty clear.

Architecture

There is no model call in this pipeline. I built the detector, ledger, and checks as a set of Python rules operating on explicit state. The same input produces the same result every time.

The ledger checks for five failure modes. Each one is derived from the records already stored in the ledger. Nothing comes from an agent's own report, and there is no manual judgment involved.

Component 1: The Commitment Detector

I could have used an LLM to extract commitments, but that would add another moving part to the experiment. The detector would depend on a model call, and the results would be harder to reproduce. I wanted to test the ledger itself, so I kept commitment detection deliberately small.

The detector recognizes four forms:

The verb must come from a fixed 13-word list:

add, create, delete, implement, modify, remove, rename, refactor, update, fix, test, document, review

The object also has to match one of a small set of concrete patterns:

Anything else is NOT_A_COMMITMENT.

There is no semantic fallback. The detector does not try to work out what the person probably meant.

The core match is:

That strictness has a cost. "I'll handle the pricing stuff" is a real commitment to a human reader, but the detector rejects it. I don't want the detector to guess what "handle" means or decide whether "the pricing stuff" is specific enough.

So I kept that part strict. If the sentence doesn't match the rules, it is rejected. That makes the detector less flexible, but I can test exactly what it accepts and why.

Testing the detector before trusting it

Before using the detector in the coordination test, I wanted to check how it behaved on cases where the answer wasn't obvious. I made a 150-row hand-labeled test set: 60 commitments covering all four sentence forms and all six object types, plus 90 negative cases across six categories.

I also included this pair on purpose:

They are almost identical. The only real difference is the object. The first one matches the dotted-path rule. The second one doesn't.

The detector got all 150 rows right:

• True positives: 60
• False positives: 0
• False negatives: 0
• True negatives: 90
• Precision: 100%
• Recall: 100%
• False-positive rate: 0%

Those numbers are only for this test set. I created and labeled the cases myself, so they show that the detector follows the rules I gave it. They don't tell me how it would perform on arbitrary coding-agent conversations. I don't have that dataset, so I'm not making that claim.

The bug adversarial testing found

The 150-row test set was clean, so I tried cases that weren't in it. That exposed a real problem in the dotted-path rule.

The rule was too loose. It accepted repository.pyz, which looks like a file path but has a questionable extension, and a.b.c.d, which is made up of four very short segments.

I tried tightening the rule. That fixed those cases, but it also rejected models.item, which was already a valid positive case in my test set.

This is where the rule hits a limit. There isn't enough information in the string itself to know whether models.item is a valid module path or whether repository.pyz is a typo. Both are just lowercase dotted strings.

So I tested both versions:

I kept the permissive version.

For this detector, rejecting real commitments is the worse failure. A path such as services.export or config.defaults can be perfectly valid even though the last part is short.

So this isn't something I tried to hide with another rule. The limitation is documented in the code, and the detector keeps the version that preserves those valid cases.

Component 2: The Ledger

The detector produces one event: COMMITTED. After that, I keep a history for each commitment instead of storing one status value and replacing it each time something happens.

I chose a history because I don't want a later event to erase an earlier one. If the ledger only stored the current status, a bad update could replace information that I still need to understand what happened.

That actually happened in my first implementation.

When the conflict check found a duplicate claim, I was marking both commitments as CONFLICTED. That meant Agent A's original commitment could become conflicted just because Agent B later claimed the same file.

That's wrong. If Agent A had already reached REPORTED, Agent B's later claim shouldn't change A's history. I fixed the check so only the new claim is marked CONFLICTED.

The ledger records what happened to each commitment. It doesn't go back and change an earlier event because another agent made a later mistake.

I use five rules to find coordination problems in those histories:

The ledger doesn't judge the code itself. It only tracks what happened to the commitments and the coordination around them.

Component 3: The Test Case

I wanted a task that was small enough to follow by hand but still had real dependencies, so I used a minimal item catalog:

There are two agents and no pre-assigned ownership. They have to decide between themselves who takes each piece of work. That's intentional. If I assign each file to an agent beforehand, the main conflict I'm trying to test disappears.

I wrote two transcripts for the same task.

In the first, the agents coordinate through chat only. In the second, they check the existing commitments before taking a task. The code that processes both transcripts is exactly the same. The only difference is what the agents do in the two cases.

These are hand-written fixtures. They aren't transcripts from a live multi-agent model run, and they aren't a statistical sample. This is one worked example designed to exercise the ledger's rules, not a benchmark.

Results: The Actual Output

This is the terminal output from evaluator.py for both transcripts. I didn't change the code between running the evaluator and writing this section.

Side by side:

Both runs register the same five commitments. C1 doesn't appear in any failure list. C2 through C5 show where the two conditions differ.

The ledger did not reduce the total number of commitments. It changed how three coordination problems were detected in this particular transcript.

What Changed With the Ledger

Three numbers changed in the comparison: conflicted commitments, dependency-notification findings, and rework.

Conflict, and the C2 / C5 pair

The conflict is the easiest one to see. Both agents claim the same file:

In the chat-only transcript, Agent B doesn't know that Agent A is already working on the file:

Agent B: I'll implement repository/items.py.

Agent A: Wait, I already finished that one. I posted about it a while back.

Agent B: Oh, I must have missed that. I went ahead and built my own copy of it too, just in case.

With the ledger, Agent B checks the existing commitments first:

Agent B: I see repository/items.py is already being handled by Agent A.

Agent B: I'll implement services/pricing.py.

The agents and the task are the same. The difference is that the second agent has something to check before making the claim.

The ledger also keeps C2 and C5 separate. C2 stays in its original history. C5 is the one marked CONFLICTED. A later duplicate claim doesn't change what happened to the first commitment.

Dependency not notified: C3

In the chat-only run, Agent B starts work on /items even though services/pricing.py has not been claimed. The dependency check catches that when /items reaches IMPLEMENTED.

In the ledger run, Agent B checks the existing commitments before taking the task, so the dependency condition is satisfied and the finding doesn't appear.

Rework

The rework finding comes from the same conflict. C5 reaches IMPLEMENTED and is then abandoned after the duplicate work is discovered.

In the ledger condition, that duplicate claim doesn't happen, so this rework doesn't appear in the report.

These three findings are all about information that wasn't available at the right time. Who already claimed the file? Has the dependency been handled? Is another agent already working on the same thing?

That's the part the ledger is designed to keep visible.

Where the Ledger Didn't Help

The two numbers that didn't change are just as important:

C4 is Agent B's commitment to write tests/test_catalog.py. It stays at COMMITTED in both runs. The ledger records the commitment, but nobody starts the work. Nothing in the ledger can make an agent pick up a task it has ignored.

C2 is Agent A's work on repository/items.py. It reaches REPORTED but never reaches VERIFIED. The ledger records what Agent A reported. It does not independently inspect the code or run tests to decide whether that report is true.

That is the boundary of this system.

The ledger helped with the coordination failures in this example that came from missing shared state. It did not change whether an agent followed through, and it did not verify whether reported work was actually correct.

So I wouldn't describe this as a system that makes coding agents reliable. The result is narrower: making commitments explicit can help with some coordination failures, but it doesn't solve execution or verification.

These are different problems. The ledger handles the first one. The second needs something outside the ledger.

What This Does Not Prove

This is one worked example. I designed it to exercise all five detection rules, so it isn't a benchmark or a sample from many independent runs.

That means I can't use these results to say how often real coding agents run into these problems, or how much a commitment ledger would help in a larger system.

The detector's 150-row result has the same limitation. It is a regression test against a hand-built labeled set. It shows that the detector followed the rules on those cases, but it doesn't tell me how well those rules would hold up on real agent conversations.

There are also other ways to design a commitment ledger, and these five failure modes are only the ones I chose to track here. They aren't a complete list of everything that can go wrong when multiple coding agents work together.

If I Were Deploying This For Real

Before I put this into a real multi-agent coding pipeline, there are three things I'd change.

First, VERIFIED needs to come from something outside the ledger. That could be an actual test run, a diff review, or some other independent check. Right now, the ledger trusts the events it receives. That's enough to demonstrate the state machine, but it isn't enough to catch an agent claiming that something works when it doesn't.

Second, a missed commitment needs some kind of follow-up. In the current example, C4 can stay at COMMITTED forever unless someone notices it. A real system could flag commitments that have been sitting untouched for a certain number of turns or a certain amount of time. The exact threshold would depend on the workflow.

That's different from simply recording the commitment. The ledger tells me that a promise exists. Something else has to make sure it gets attention.

Third, the dependency map should come from the codebase instead of being written by hand. Hardcoding five relationships is fine for this example. It wouldn't hold up once the project gets bigger. Real codebases have dependencies that change as the code changes, so maintaining that graph manually would quickly become another source of stale information.

None of these changes are particularly complicated. I left them out because they would have changed what I was testing.

The question here was narrower: does making a commitment explicit change coordination outcomes?

In this example, it did for failures caused by missing shared state. It didn't help with failures caused by an agent not following through.

Design Decisions and Limits

Status comes from events, not more language parsing. After COMMITTED, the states STARTED, IMPLEMENTED, TESTED, REPORTED, and VERIFIED come from events passed to the ledger. I didn't add another grammar to read agent messages and guess when one of these things happened. That would make the result depend on another layer of interpretation.

One commitment per line. The detector only looks at the beginning of the text it receives. So a sentence like "Repository and pricing both look done, so I'll implement /items now" doesn't get registered. I kept the transcripts in a form where the commitment appears on its own line rather than adding another parser for cases like this.

That does leave a gap. An agent can make a commitment in the middle of a longer sentence and the detector will miss it. Handling that properly would mean figuring out where the actual sentence or clause starts before the commitment detector even gets involved.

Commitment renegotiation isn't modeled. An agent might start with one task and later change the scope. This version doesn't have a state for that. Once the commitment is recorded, the ledger can track what happens to it, but it doesn't understand a conversation where the original commitment is changed.

I only tested two agents. The conflict check itself doesn't depend on there being exactly two. It looks for another agent with an active commitment on the same object. But I haven't tested the system with three or more agents, so I don't want to assume the same results would hold there. More agents also introduce dependency chains that this version doesn't model.

Takeaway

Conversation isn't enough when multiple coding agents are working on the same codebase. A small persistent record of commitments can make some coordination problems visible before they turn into duplicate work, conflicts, or dependency mistakes.

But recording a commitment doesn't mean the work gets done. It doesn't mean the result is correct either.

A commitment ledger can improve coordination state. It cannot create execution discipline.

That's the limit of what I showed here. The ledger helped with the problems caused by missing shared state. It didn't solve the problems caused by agents not following through or by work that still needed independent verification.

Resources

Code and data: Full source, the 150-row labeled detector set, both example transcripts, and the complete test suite (12 tests) are available at https://github.com/Emmimal/commitment-ledger/. The detector, ledger, and evaluator use only the Python standard library. The test suite uses pytest.

References

[1] Khatua, A., Zhu, H., Tran, P., Prabhudesai, A., Sadrieh, F., Lieberwirth, J. K., Yu, X., Fu, Y., Ryan, M. J., Pei, J., & Yang, D. (2026). CooperBench: Why Coding Agents Cannot be Your Teammates Yet. arXiv preprint arXiv:2601.13295. https://arxiv.org/abs/2601.13295

[2] CooperBench project page and benchmark code. https://cooperbench.com/ and https://github.com/cooperbench/CooperBench

[3] Python Software Foundation. (2024). re — Regular expression operations. Python 3.12 Standard Library Documentation. https://docs.python.org/3/library/re.html

[4] Python Software Foundation. (2024). dataclasses — Data Classes. Python 3.12 Standard Library Documentation. https://docs.python.org/3/library/dataclasses.html

Disclosure

All code in this article is my own original work, developed and tested on Python 3.12 (Windows). The system itself — detector, ledger, and evaluator — uses only the Python standard library. The test suite uses pytest. No API key, network access, database, vector store, or external model call is required at any point.

Every terminal output block and result table in this article is taken directly from an actual run of the delivered codebase, unedited, and was independently reproduced on a second machine before publication.

All diagrams in this article, including the featured image, were created by the author. The featured image was generated with ChatGPT (DALL·E).