Vibe Patenting: LLM Judges Improve AI Patent Drafting Quality

A significant finding is that iterative feedback allows a low-reasoning agent to perform nearly as well as a much more expensive high-reasoning agent. This suggests that effective evaluation loops can reduce computational costs without sacrificing output quality. Stronger base models and increased reasoning capabilities generally lead to better drafting, but domain-specific agentic workflows provide additional gains.

To validate the LLM judge, the researchers compared its assessments with those of an independent professional patent attorney. While there was meaningful agreement, the study also identified systematic calibration differences that depend heavily on the specific metrics used. These findings highlight both the potential utility and the current limitations of using LLMs as evaluators for complex, high-stakes professional tasks.