SymCE corpus released for Counterexample generation
SymCE is a new dataset of 4,707 false undergraduate‑algebra and real‑analysis conjectures, each paired with an executable Python verifier. The authors train Qwen3‑4B and Gemma‑3‑4B, showing that counterexample‑only supervised fine‑tuning drops true‑theorem recognition from 0.27 to 0.00, while reinforcement learning with a sparse outcome‑only reward restores it to 0.66.
Key points
- SymCE: 4,707 false undergraduate‑algebra and real‑analysis conjectures with executable verifiers.
- Counterexample‑only SFT drops true‑theorem recognition from 0.27 to 0.00; RLVR with sparse reward restores to 0.66.
- 4B model outperforms 7B open‑weights math specialists and matches six frontier commercial APIs.
The 4B model outperforms every evaluated 7B open‑weights math specialist and remains competitive with six frontier commercial APIs. A human audit of 177 verifier decisions finds 97.7% accuracy, and the model transfers under unchanged prompting to GSM8K, MATH‑500 and MMLU‑college‑math.
These results illustrate how reinforcement learning can repair imitation failures and provide a large, verifiable benchmark for training models to generate counterexamples in theorem proving.
The story so far
3 episodes →- SymCE corpus released for Counterexample generationthis story
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Google VP Yossi Matias says AI’s biggest impact may come from intersecting fields · 1 src
- Researchers adapt speech language model for simultaneous translation using prefix supervision · 1 src
- Study finds inductive prompting most consistent for LLM generalization in temporal extraction · 1 src
- AraBERT-based framework reaches 96.88% accuracy on Arabic DP ambiguity · 1 src
- Researchers introduce APDMem hierarchical memory for long-context LLM assistants · 1 src
Comments
via GitHub Discussions