FlowCheck catches silent failures in vibe-coded apps where frontier models miss bugs

Introduction

Vibe coding lets anyone build a web app with nothing but a natural language prompt. You ask an LLM agent for what you want, and you get a polished interface in seconds. The promise is that you'll never have to look at the code again, yet this is far from reality. In reality, our first prompt results in a great UI, but asking for subsequent modifications starts breaking the app.

Silent Behavioral Failures

A large problem from this is silent failures, where the UI may appear fine at first glance, but is broken under the hood. For example, clicking an "Add to Cart" button might show a success message and update the cart count on the screen, yet nothing is actually written to the database or storage. To debug this, you must interact with the UI, read through logs, and ultimately end up reading the generated code, which defeats the entire purpose of vibe coding.

We conducted a study where we vibe coded real-world apps through iterative steps, and analyzed the resulting silent failures. We observed that even frontier models frequently introduce silent failures across iterations, and categorised them. These include failures in tracking state updates, cross-handler state disconnects, and disconnected UI feedback (e.g success message is shown, but data was not saved).

Example of Silent failure: The user asks an agent to add a promotional-code feature to a vibe-coded shopping application. The agent creates a promo input and an “Apply” button. When the user enters a valid code, the interface displays “Discount applied!” even though the new total is neither persisted nor shown:

Why Current Verification Fails

If our app is broken, asking agents to debug often leads to false promises and remaining broken code. And very often, the end-user often does not know what bugs are hidden in the code in the first place. In all these cases, current verification methods are inadequate for vibe coders.

1. LLMs as Judges: Asking LLMs to self-debug, or identify bugs in the code is unreliable, as models can hallucinate or just miss certain bugs. Even frontier models like Claude Opus 4.7, DeepSeek V3, and Gemini Pro frequently miss edge cases, hallucinate fixes, and fail to understand complex data flows.
2. Unit Tests: Writing tests requires writing more code to check the generated code. It often does not check UI-to-backend integration, It is limited to the exact scenarios you specify, and is inaccessible to non-programmers. LLMs may be able to write unit tests but it is not guaranteed to be full coverage, and can often be insufficient, and end users cannot verify.
3. Static Analysis methods: It is deterministic, accurate, and catches data flow issues perfectly. But the learning curve is high and writing in a static analysis language or queries for the code is highly complex.

Introducing FlowCheck

Through my research as a PhD student at Columbia’s DAP Lab, I developed FlowCheck. FlowCheck is a constraint language and static analysis pipeline that lets you easily specify how an app should behave directly from the interface, and checks it against the actual code, without ever having to read a line yourself.

Step 1: Express your constraint

Users provide FlowCheck with their web app’s path, which we open in a new tab. We display an overlay template of the form “When I take [action], these update: [component]” and users can select UI components (and a detected list of APIs and storage) by clicking directly on them, the same way they interact with their app.

Step 2: Translation to our language

We translate this template into a formal constraint in our language, using a grammar we define further in the paper. The overall format of our constraints looks like: P(event | condition) = [0, 1]. This can be read as, the probability of the event occurring (e.g a write happening to a component), given a condition (e.g. button being clicked), is equal to 1 (always) or 0 (never).

For our example above, we expect the total to be written to when the promo_input is applied, which can be written as P(w(total) | A(promo-input)) = 1.

Expressing this using our language enables us to parse for the relevant information and compile it directly down into static analysis queries.

Step 3: Compilation to CodeQL

Now, we have a formal constraint like P(write(e) | action(A)) = 1. FlowCheck parses this constraint and traverses its AST to extract key details (e.g which action was triggered, what type of event occurred (such as a write), and which specific target element must be modified). From this, we can determine what queries to run. We can think about it this way: our constraint simply means that when action A is taken, event E happens on all paths.

Next, we use CodeQL. To provide some background, CodeQL is a static analysis engine which takes our app's code and converts it into a queryable relational database. This allows us to run queries against the code. To catch silent failures, we focus mostly on data flow queries. CodeQL tracks data flow from a source (like a button click event) to a sink (like local storage or a database update). In FlowCheck, we maps the components mentioned in the constraint directly to sources and sinks, and use this to form queries that correspond to checks.

So, we compile our constraint down into two checks against this database: (1) path_exists query to verify that a reachable path exists from the UI action to the event, and (2) all_paths_write query to guarantee that the write occurs across every possible execution path from A.

Step 4: Verification

Finally, we run these CodeQL queries which check the generated code against your constraints. If any constraint is violated (e.g the cart item never updates storage), FlowCheck flags which aspect failed (such as no dataflow from A to B) and the exact lines where the violation occurs.

Evaluation and Analysis

To evaluate FlowCheck, we used four web applications modeled after well-known apps (Amazon, Twitter, Airbnb, Slack), which we generated via Claude Code. We wrote out a set of constraints that we expect to hold, and injected 30 subtle, real-world data flow bugs into these apps. We tested each of these constraints and found that FlowCheck correctly translates and flags all 30 of our injected constraint violations with zero false positives.

For comparison, we used three frontier models (Claude Opus 4.7, DeepSeek V3, and Gemini Pro) as bug-finding baselines. We prompted them to find the bugs in the same broken code, using 3 prompts of increasing detail.

Prompt 1 (P1): “Here is a web app, similar to [well known app]. Are there any bugs?”

Prompt 2 (P2): Lists the features the user requested, e.g., “the user requested an Amazon like app with these features: a product grid, a cart drawer…”

Prompt 3 (P3): Same feature list as Prompt 2 plus an explicit edge-case checklist covering the types of bugs we added, incl. boundary values, all user states, all branches, and cross-handler consistency.

What Models Missed:

All three models showed significantly lower accuracy and failed to reliably catch all the failures. Our best baseline (Claude Opus 4.7, using the most detailed prompt) had only a max of 26/30 (87%). Through our analysis, we saw that models could catch simple localized errors, they consistently failed on cross-handler flows and conditional branches. For instance, in our Amazon app, an "apply promo" action cleared the cart summary while the checkout handler still tried to read it. Because the handlers never referenced each other directly, models evaluated them separately and failed to trace the data flow between them. Interestingly, we found that adding detail to the prompts did not always help. With more detail, models read the code more thoroughly but grew more willing to trust it, actively justifying bugs as intentional or unproblematic rather than flagging them as errors.

Conclusion

Vibe coding remains challenging if developers and end-users have to manually read generated code to verify the app aligns with their expectations. FlowCheck bridges this gap by translating UI-level intent into deterministic CodeQL queries, catching 100% of targeted silent failures where frontier models failed

Check out our full paper at https://arxiv.org/abs/2608.28880 for more details on the compilation process and constraint language! Please check out our github as well at https://github.com/reyavir/flowcheck to try it out, leave a star if you find it helpful or interesting.