Gemma 4 graph optimizations improve prompt processing performance
A pull request by Max Krasnyansky to the llama.cpp repository introduced optimizations for Gemma 4’s graph processing. The changes, picked on November 2, 2026, focus on prompt processing, including pruning, mask cropping, and speculative decoding improvements. The updates aim to reduce unnecessary tensor copying and memory overhead during multi-layer processing, particularly when backends differ…
Key points
- Pull request by Max Krasnyansky optimizes Gemma 4’s graph for prompt processing on November 2, 2026
- Changes restore `ggml_cont()` to reduce tensor copying and memory overhead across backends
- Speculative decoding logic refined to avoid redundant hidden-state extraction in shared-layer attention
The PR highlights ongoing work to further optimize microbatch processing and post-KV-cache operations. Key changes include pruning unused prompt rows before shared layers, avoiding redundant hidden-state extraction for output-free prompt chunks, and preserving KV prefix embeddings. The goal is to maintain performance gains in speculative decoding while preserving existing MTP and classic speculative decoding behavior. The pull request references co-assistance from Claude (Opus 5.5) for conflict resolution and notes unresolved tasks like refining speculative-simple mode for shared-KV MTP.
Model page: Claude Opus 5.5 →
gemma4: graph optimizations (massive gains on pp for E2B and E4B) by max-krasnyansky · Pull Request #31356 · ggml-org/llama.cpp
github.com · 29 October 2026Loading the full article…
This text was published by github.com. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
Coverage and discussion
1source- Reddit discussionreddit.com
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Anthropic releases FlashProve for cryptographic AI inference proofs · 4 src
- Ai2 releases StateBench to test long-context state tracking in models · 1 src
- MIT researchers report GPT-6 Astra natively reads and writes neural network weights, raising Qwen3.5-0.8B MATH-500 from 7.9% to 48.1% by editing 0.016% of weigh · 6 src
- SETI’s Allen Telescope Array uses AI agents to speed up signal analysis · 1 src
- Stanford study: LLMs' claims expire quickly · 1 src
Comments
via GitHub Discussions