LLaMa Adds Maple 20B-A1B MoE Architecture with Ternary Quantization

Conversation

I don't see ternary TQ1_0/TQ2_0 being applicable, since they are group size 256, while the model was trained on 128. We still don't have 128 group side ternary, so only q2_0 will work.

Thanks for your help on this PR @Green-Sky @AlexGabbia! Yeah I think any group size that is divisible by row dim will work as the model was trained with row wise scales. The default 256 group scales should work 👍 .

Thats great, thanks for noting that.

Btw, have you considered adding more sparsity via things like ngram embeddings (longcat) or per-layer embeddings (gemma). Would be easy ways to add 5-10B without hurting inference speed much. Ideally even in ternary or binary. :)

Ran the full wikitext-2 test split this time (not just 20 chunks). All 4 official DeepGrove GGUFs, CUDA on a RTX 5070 Ti, -ngl 99 -t 4. Branch rebased on latest master.

TQ1_0 and TQ2_0 give identical PPL — same weights, different packing, makes sense.

Compared to your q2_0 numbers @Green-Sky: tq2_0 scores lower (79.76 vs 86.92 @512, 61.14 vs 64.83 @2048). Probably because TQ2_0 keeps the original ternary weights as-is while q2_0 re-quantizes them.

head-Q4_K costs ~1.5 PPL @512, ~0.7 @2048 over head-F16.

@2048 is much lower than @512 across the board — Maple's SWA-512 needs the context.

Compared to your q2_0 numbers @Green-Sky: tq2_0 scores lower (79.76 vs 86.92 @512, 61.14 vs 64.83 @2048). Probably because TQ2_0 keeps the original ternary weights as-is while q2_0 re-quantizes them.

Hm, I wonder whats wrong here, but the q2_0 weights themself are lossless. I made sure by requantizing them back to tq1_0 and the 2 files are hash identical.

54016e4d543bd688829e67103fc85b8396db94b7f8eb3f81fa95884e44393872  maple-preview-TQ1_0-head-Q4_K.gguf
quant to q2_0 ->
3bb9bd3e3755d06810b98f29a2edb0127199dc1b55a40a3c306cd69ac5615e1e  maple-preview-q2_0-head-q4_K.gguf
quant to tq1_0 ->
54016e4d543bd688829e67103fc85b8396db94b7f8eb3f81fa95884e44393872  maple-preview-requant_tq1_0.gguf

So the divergents needs to come from the ops running in different quantizaitons.

But not sure, the gap is rather large. Not sure different quant + different backend account for all that.

Maple's SWA-512 needs the context.

This reads wrong, slap your agent writing for you on the wrists.

Fair point on the phrasing — corrected: the lower PPL @2048 vs @512 reflects the global-attention layers (every 4th layer) capturing long-range patterns that SWA-512 layers miss at short context. My agent earned the wrist-slap :D

On the divergence: you're right that the weights are lossless — I reproduced your hash check. So the gap (79.76 TQ2_0 vs 86.92 q2_0 @512) must come from the matmul ops running in different quantizations and/or backends (CUDA vs CPU).

To isolate the variable: I'll run TQ2_0 on CPU (same backend as you) and q2_0 on CUDA (same backend as me). That should tell us whether it's the backend or the quant type causing the gap. Will post results when ready.

I reproduced your hash check — q2_0 → TQ1_0 round-trip is bit-identical. So the divergence isn't in the weights. To isolate the variable, I ran the same q2_0 file (head-Q4_K embd-f16, 6060.33 MiB / 2.51 BPW — matches your numbers exactly) across backends on the full wikitext-2 test split, same -c 512 -b 512:

80.78****86.92 On x86, q2_0 is consistent across backends (CUDA 80.81 vs CPU 80.78 — 0.03 gap). Same for TQ2_0. But your 86.92 is 6 PPL higher than my x86 CPU run on the same file.

The only variable left is CPU architecture. Are you on ARM/Apple Silicon? If so, the gap is in the q2_0 ARM kernel, not in the model or the quant type — and TQ2_0 native is the numerically stable path.

Happy to help debug the ARM kernel if you can share a repro. The architecture itself is simple enough that a standalone test should pinpoint it.

(Fixed the SWA phrasing btw — the @2048 improvement comes from the global-attention layers, not from SWA "needing" context.)

What I did was CUDA (RTX 2070), I see now that I forgot to mention this.

Ok lets sync up the launch command, to make sure the issue is not there.

$ llama-perplexity -m models/maple-preview-q2_0-head-q4_K.gguf -f wikitext-2-raw/wiki.test.raw -c 512

For f16 embeddings.

I also checked with -b 512 and it results in 86.9592 +/- 0.89297.

Also ran $ test-llama-archs -a maple

main: using seed 36176602
|     Model arch.|                                Device|Config|   NMSE vs. CPU|Roundtrip|
|----------------|--------------------------------------|------|---------------|---------|
|           maple|               NVIDIA GeForce RTX 2070|   MoE|  OK (5.14e-13)|     SKIP|
|           maple|AMD Ryzen 9 PRO 3900 12-Core Processor|   MoE|  OK (0.00e+00)|     SKIP|
|           maple|                                  Meta|   MoE|  OK (5.14e-13)|     SKIP|

Which has even less difference between cpu and accelerator than your setup.

I rand ppl on the full test split using the provided tq1_0 head-q4_k gguf on my cpu.

-f wikitext-2-raw/wiki.test.raw -c 512 -b 512

Which falls in line with my other values.

173c87a53759e0201f33e0ccf978e510c2042d7f2cb78229d9a50d79b9e7dd08 wikitext-2-raw/wiki.test.raw

Ok, would be great to have this done before the finished model releases.

Would be great if someone else also can run ppl numbers and tests, because its kinda stuck right now.

Also @AlexGabbia pls rebase.

Add MODEL_ARCH.MAPLE, its "maple" name, and the tensor list for the Maple 20B-A1B ternary MoE architecture: token embeddings, output, attention with Q/K RMS norms, and per-expert FFN tensors.

Register MapleForCausalLM in the HF architecture map and add the converter for the Maple 20B-A1B ternary MoE model: 24 layers, 256 experts with 8 active, sliding-window attention (SWA-512) interleaved with global attention at a 3:1 ratio, partial rotary factor 0.5, and per-expert weight stacking into merged 3D tensors.

Add the Maple 20B-A1B ternary MoE architecture: 24 layers, 256 experts with 8 active, sliding-window attention (SWA-512) interleaved with global attention at a 3:1 ratio, and ternary TQ1_0/TQ2_0 quantization support.

• register LLM_ARCH_MAPLE between MAMBA2 and JAMBA
• implement llama_model_maple: Q/K RMS norms after projection (GEMMA4 style), rope applied only on SWA layers (nope_on_global_attention), ISWA KV cache, and MoE FFN with swiglu gate clamp at +7 (DEEPSEEK4 style)
• mark MAPLE as unsupported by the model saver (roundtrip skipped)

- load_arch_hparams: use n_ff_exp_arr + n_ff_exp() accessor (upstream
  changed these from a scalar member during the rebase)
- sliding_window_pattern: get_arr, the pattern is mandatory for this arch
- partial_rotary_factor: read only from rope_parameters (base.py mirrors
  the top-level key automatically)
- document why TOKEN_EMBD/OUTPUT are forced to F16 (they are the two
  dense tensors in Maple, and the reference GGUFs ship them as F16)
- add @ModelBase.example("deepgrove/maple-preview")

get_arr for maple.attention.sliding_window_pattern requires an array, but

the harness only emitted a per-layer array for the arches in its list, so test-llama-archs -a maple failed to load the model. Assisted-by: DeepSeek Harness

@CISC @Green-Sky rebased on current master.

All six review comments are in. maple.py:14, maple.py:27, maple.cpp:7 and maple.cpp:10 are applied as suggested.

On maple.py:43: in the official TQ2_0 head-F16 GGUF the ternary part is 168 tensors, and TOKEN_EMBD / OUTPUT are the only dense tensors with real weight mass (2048 x 151936 each). Everything else dense is norms and the router, which is already F32. F16 on those two matches what DeepGrove ships. Happy to drop it to super() if you would rather have the default.

On maple.cpp:17: the converter does not write the key because config.json has no clamp value to convert, unlike deepseek or hy_v4 where it comes from config. So the fill is the fallback, and the official GGUFs carry maple.swiglu_clamp_exp = [7.0] * 24 anyway.

One thing worth flagging: maple.cpp:10 as suggested broke the arch test. The harness only emits a per-layer array for the arches in its list, so it failed with array key not found in model: maple.attention.sliding_window_pattern. I added LLM_ARCH_MAPLE to that list, same as dots3note which uses the same call. test-llama-archs -a maple passes now on the rebased branch, 8.99e-08 on GPU and Meta, 0.00e+00 on CPU, and the full suite passes too.

@Green-Sky the PPL gap was my wikitext file. Thanks for posting the hash, that was the missing piece. Yours is 173c87a5..., which is what scripts/get-wikitext-2.sh downloads. Mine has CRLF line endings and 2890 extra line breaks, so it tokenizes differently. Same model, same machine, same backend, only the file changed:

wiki.test.raw That is 5.77 PPL from the file alone. Against your 86.96 on head-Q4_K: the Q4_K head costs me +1.39 over F16 on the same file, so on the upstream file I would expect about 86.58, a 0.38 residual. That is about your own CPU vs CUDA spread, so there is no kernel bug here. My mistake, and I will redo the rest of the table on the upstream file.

Also correcting my earlier table: the row I marked CUDA for the TQ2_0 file was not actually running the ternary matmuls on the GPU. It took 20 min against 1 min 18 s for the real CUDA run.

CUDA stays out of this PR, the description already says CPU-only. For the record I have the TQ2_0 dequantize path and the MMVQ vec dot fixed locally and will open that separately.

The loader prefilled 7.0 and read the key optionally. The converter now writes it and the loader reads it as required, because llama-graph.cpp skips the clamp when the limit is 0 and an optional read would silently run unclamped. The test harness provides the key for the same reason. Also drops tensor_force_quant: base.py already forces FFN_GATE_INP to F32 and TOKEN_EMBD/OUTPUT to F16 for ternary file types. Assisted-by: DeepSeek Harness

@CISC both applied.

conversion/maple.py:49 - removed. You are right, base.py:1057 already forces TOKEN_EMBD and OUTPUT to F16 when the ftype is TQ1_0 or TQ2_0, and base.py:1023 already forces FFN_GATE_INP to F32, so the whole tensor_force_quant override was dead weight. The four official GGUFs match that: token_embd is F16 in all four, ffn_gate_inp is F32 in all four, and output is F16 or Q4_K depending on the variant.

src/models/maple.cpp:19 - moved to maple.py:set_gguf_parameters(), and the loader now just reads the key. I made the read required rather than optional on purpose: llama-graph.cpp:2227 skips the clamp entirely when the limit is 0, so an optional read would silently run unclamped and drift from the reference. That needs the test harness to supply the key, the same way it already does for hy_v4, deepseek4 and bailingmoe3, so I added a small block there. All four official GGUFs carry maple.swiglu_clamp_exp = [7.0] * 24, so nothing in the wild breaks. Say the word if you would rather have it optional.

test-llama-archs -a maple passes on the new head (8.98e-08 on GPU and Meta, 0.00e+00 on CPU), and an official GGUF still loads.

@Green-Sky here are the numbers on the upstream wikitext file, all four official GGUFs, CPU -ngl 0 -t 16, -c 512 -b 512, full test split:

Your tq1_0 head-Q4_K on the same file and the same backend is 87.3064 +/- 0.89495, so the like-for-like pair is 87.0225 against 87.3064. That is a 0.28 residual, the same order as the run-to-run spread I see here (the two F16 variants differ by 0.33 even though TQ1_0 and TQ2_0 represent the same ternary weights).

For completeness, with the local CUDA TQ2_0 dequantize path, TQ2_0 head-F16 gives 85.1846 +/- 0.87378 on the GPU, 0.023 away from the CPU number on the same file.

So the gap was the wikitext file, and it is closed. Thanks for pushing on it.

@Green-Sky @CISC one small thing that would help, and it is not about this PR's scope.

CUDA support for TQ2_0 is ready on a branch, draft PR #28769. The one gap I cannot close myself is AMD: HIP compiles in CI, but TQ2_0 is never exercised numerically there, because ci/run.sh runs test-backend-ops with -b CPU only and test-llama-archs uses F16 tensors. If anyone has an AMD GPU and a few minutes:

test-backend-ops -o MUL_MAT,MUL_MAT_ID -p TQ2_0

That is 17 cases and would settle it. Thanks.

ty flagged the stack() closure: it takes no argument, while LazyBase is annotated with func: Callable[[Any], Any]. Pass the tensor list through args instead of closing over it, the same way kimi_k3 does, so the callable shape matches. Assisted-by: DeepSeek Harness

@CISC fixed in 0d0b53c.

ty was right: stack() took no argument while LazyBase is annotated with func: Callable[[Any], Any]. The tensor list now goes through args instead of the closure, the same way kimi_k3.py does it. I verified with ty 0.0.80 before pushing, and reverted the fix once to confirm the checker really does flag the original.

For the record on the other four failures: ubuntu and both gpu-webgpu jobs fail on artifact and model downloads, and their ctest runs report 100% passed. The openvino one is SWIGLU_CLAMP at ERR 3.07e-7 against a 1e-7 tolerance, which is inside the op itself - it is already on master, and this PR only adds LLM_ARCH_MAPLE to its arch list.