Hugging Face proposes Model Integration Contracts to catch regressions in Transformers
Hugging Face introduced Model Integration Contracts (MICs), a testing standard that uses shrunken model fixtures to catch regressions in the Transformers library before they reach the main branch. The approach was motivated by a bug in UMT5's decoder that went undetected for nearly two months after PyTorch's scaled dot-product attention was enabled on July 30, 2026. A small UMT5 fixture, using…
Key points
- UMT5 bug from July 30 to Sept 28, 2026 caught by 66 MB fixture vs 1,227 MB original
- MICs include fixture, usage path, and human-reviewed baselines; stored in huggingface-mic
- tmic CLI automates contract creation; MIC jobs report but don't block PRs as of Nov 1, 2026
MICs consist of three parts: a fixture (a minimized model preserving all layer types), a usage path (a runnable example like text generation or a training step), and baselines (human-reviewed expected outputs). The contracts are architecture-specific, not checkpoint-specific, and are stored in the huggingface-mic repository with pinned revisions for reproducibility. A CLI tool called tmic automates fixture creation, usage drafting via LLM, and baseline recording. As of November 1, 2026, MIC jobs report failures in Transformers CI without blocking pull requests, pending measurement of real-world bug detection rates, false alarm frequency, and review overhead.
The story so far
2 episodes →- Hugging Face proposes Model Integration Contracts to catch regressions in Transformersthis story
Transformers Model Integration Contracts
Hugging Face · 3 November 2026
Loading the full article…
This text was published by Hugging Face and written by Tarek Ziadé. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- OpenAI releases PaperBench benchmark for AI research reproducibility · 1 src
- Thoughts on LLMs Editing Weights: What's New? · 1 src
- MIT researchers report GPT-6 Astra natively reads and writes neural network weights, raising Qwen3.5-0.8B MATH-500 from 7.9% to 48.1% by editing 0.016% of weigh · 4 src
- Simon Willison sends October sponsors-only newsletter · 1 src
- Anthropic releases FlashProve for cryptographic AI inference proofs · 1 src
Comments
via GitHub Discussions