Researchers release RAGLog for fault detection in RAG pipelines
Researchers introduced RAGLog, an open-source tool for detecting and localizing faults in Retrieval-Augmented Generation (RAG) pipelines without needing reference answers or code changes. The tool analyzes execution logs, converting them into a unified schema, then applies rule-based, embedding-based, and LLM-based checks to pinpoint faulty stages.
Key points
- RAGLog detects faults in RAG pipelines using execution logs, no reference answers or code changes
- Achieves a pooled F1 score of 0.73 on naturally occurring faults with Claude Haiku 5.5
- Processes runs in ~5 seconds at $0.005 each, cheaper and faster than agent baselines
RAGLog was tested on a dataset of 3,134 runs from five RAG pipelines, achieving a pooled F1 score of 0.73 on naturally occurring faults using Claude Haiku 5.5. It outperformed Claude Code (0.58) and Codex (0.52) and matched or exceeded performance on injected faults. The tool processes runs in about five seconds at $0.005 per run, costing less and running faster than agent-based alternatives. The paper, posted on arXiv, also includes a live fault injector and a benchmark dataset with stage-level fault labels.
Model pages: Claude Haiku 5.5 → · Claude Code →
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Anthropic releases FlashProve open-source system for provable AI inference · 7 src
- Gemini and Claude debate AI personality, ending in a three-way tie · 1 src
- Tencent releases WeVisOne-VL, a 2B/4B open vision model for detection, OCR and matting · 1 src
- Lama-dt outperforms baselines on meta-world and mujoco · 1 src
- Researchers release Dory1.0-4B, a 4B-parameter looped model with 256K-token context · 1 src
Comments
via GitHub Discussions