# FlowCheck catches silent failures in vibe-coded apps where frontier models miss bugs

Digest AI · Research · published 2026-09-18T15:30:01Z

Canonical: https://digestai.news/story/flowcheck-catches-silent-failures-in-vibe-coded-apps-where-frontier-mo

## Summary

A study of vibe coding – where an LLM agent builds a web app from a natural‑language prompt – found that iterative modifications often introduce silent failures, such as UI actions that appear successful but do not update the database. The researchers evaluated four popular apps (modeled after Amazon, Twitter, Airbnb and Slack) generated with Claude Code, injected 30 subtle data‑flow bugs, and tested three frontier models (Claude Opus 4.7, DeepSeek V3 and Gemini Pro) as bug‑finding baselines. Even with detailed prompts, the best model caught only 26 of the 30 bugs (87%) and struggled with cross‑handler flows and conditional branches.

To address this gap, the author, a PhD student at Columbia’s DAP Lab, created FlowCheck, a constraint language and static‑analysis pipeline that lets users specify expected UI behavior and automatically compiles those constraints into CodeQL queries. In the evaluation FlowCheck flagged all 30 injected violations with zero false positives, demonstrating that deterministic static analysis can reliably detect silent failures that LLMs miss. The paper and source code are available on arXiv and GitHub.

## Key points

- FlowCheck translates UI constraints into CodeQL queries and flagged all 30 injected silent failures with zero false positives.
- Frontier models Claude Opus 4.7, DeepSeek V3, and Gemini Pro missed up to 4 of 30 bugs, achieving at most 87% accuracy.
- Silent failures include state‑update mismatches and UI feedback that appears correct while data isn’t persisted.

## Why it matters

Detecting silent failures in automatically generated code is essential for reliable low‑code app creation, preventing broken e‑commerce flows and costly debugging, and shows static analysis can outperform current LLM debugging approaches.

## Sources

1. [Coding Agents Keep Shipping Silent Failures](https://towardsdatascience.com/coding-agents-keep-shipping-silent-failures-here-is-how-to-catch-them) (Towards Data Science, 2026-09-18)

## Cite

Digest AI, "FlowCheck catches silent failures in vibe-coded apps where frontier models miss bugs", 18 September 2026, https://digestai.news/story/flowcheck-catches-silent-failures-in-vibe-coded-apps-where-frontier-mo

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/flowcheck-catches-silent-failures-in-vibe-coded-apps-where-frontier-mo.json