Study finds AI agents lose intent in smart contract synthesis

The study found that common grading methods, such as AI judges and standard analysis tools, often miss obvious mistakes. It also showed that checking only standard functions hid about half the performance differences between different AI pipelines. The authors propose an "elicitation-first" architecture where the agent asks the client a short checklist of questions before writing code. This approach, combined with checking against an independently written answer key, was more effective than reviewing or patching code afterwards across three AI models tested.

The paper claims this is the first study in smart contract synthesis to gather missing requirements before generation and verify results against an executable model. The authors release their tools, 600 tasks, and all results, noting that the lessons apply to any setting where AI agents turn requests into programs.