Researchers propose Policy-as-Skill for governed LLM decision-making
Researchers introduced Policy-as-Skill (PaS), a modular runtime framework designed to integrate governance, evidence validation, and auditability into large language model (LLM) decision-making. The system packages policy functions as executable, versioned capabilities to handle tasks like compliance checks, risk assessments, and operational decisions.
Key points
- Policy-as-Skill (PaS) adds governance, evidence checks, and audit trails to LLM decision-making via modular, versioned policy modules
- Tested on 600 tasks with **Gemma4**, achieving **53.8% exact accuracy** and **0.346 macro-F1** without deterministic control
- Deterministic control boosts accuracy to **61.2%** but works best for specific tasks, not universally
Evaluated on 600 development tasks using a fixed Gemma4 backend, PaS+Audit achieved 53.8% exact accuracy and macro-F1 of 0.346. With deterministic control, accuracy rose to 61.2%, though performance varied by task. The framework outperformed traditional LLM+RAG approaches on governance and review metrics, including citation precision of 1.000 and policy-reference recall of 0.984. The authors emphasize its potential for selective rule-based intervention rather than universal deterministic control.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Study finds multimodal AI models shift answers based on evidence order · 1 src
- PLLM+ pipeline solves 1,500 Python dependency issues in benchmark · 1 src
- Researchers propose method to balance conflicting AI objectives without retraining models · 1 src
- Researchers release agimud for Multi-Agent simulations with human interaction · 1 src
- Microsoft Copilot AI predicts bitcoin could reach $180,000 by early 2027 · 2 src
Comments
via GitHub Discussions