Skip to content
OmegaFactory
On This Page · Jump to Section10 sections
TECHNICAL DOCUMENTATION8 Releases · 24 GGUFs VerifiedN=200 Holdout · Seed 42

Certified GGUF Quantization Ladders & Empirical Benchmarks

How Omega Factory evaluates, verifies, and publishes 3-rung post-training GGUF quantization ladders (Q8_0, Q4_K_M, Q2_K) across state-of-the-art Mixture-of-Experts (MoE) and Dense architectures. Every checkpoint is benchmarked on held-out reasoning splits (ARC-MC, GSM8K, MATH-500) with item-level McNemar answer-flip diagnostics and cryptographic SHA-256 provenance markers—compatible out of the box with Ollama and llama.cpp.

Latest Certified GGUF Ladder Updates
Certified Releases
8 Models4 MoE · 4 Dense
Immutable Artifacts
24 GGUFsQ8_0 · Q4_K_M · Q2_K
Holdout Evaluations
32 MatricesN=200 · Seed 42 · Greedy
McNemar Diagnostics
72 Flip RowsPer-item Broke vs Fixed
01 · Quantization Architecture

The 3-Rung Certified GGUF Precision Ladder

Instead of publishing dozens of unverified quantization shards with unknown accuracy degradation, Omega Factory standardizes every public release around a strict 3-rung post-training quantization (PTQ) ladder (Q8_0, Q4_K_M, Q2_K) benchmarked directly against the unquantized FP16 or BF16 reference weights. We do not train on evaluation holdouts, ensuring zero benchmark contamination across all published checkpoints.

Q8_0~0.50x–0.53x (-47% to -50%)

Near-Lossless Reference Rung

8-bit round-to-nearest block quantization with 32 weights per FP16 scale factor (w = q × block_scale). Matches unquantized FP16/BF16 within statistical noise on every evaluated model, achieving 0.0 pp ARC-MC net delta on 7 of 8 releases (+10.0 pp on the 8th) and zero broken ARC-MC answers on 6 of 8 catalog releases.

Target: 32–96 GB VRAM · Production & Verification
Q4_K_M · Recommended~0.28x–0.33x (-67% to -72%)

Production Pareto Sweet Spot

4-bit k-quant mixed precision using 256-weight super-blocks with 6-bit sub-block scales/mins while preserving critical attention & down-projection tensors at higher bit widths. Delivers the highest accuracy-per-gigabyte efficiency and binds our canonical OMEGA_VERIFIED_PUBLIC_MODEL_V1 release marker.

Target: 16–24 GB VRAM (RTX 3090/4090 / Mac 32GB)
Q2_K~0.18x–0.21x (-79% to -82%)

Constrained Edge & Sparse MoE Rung

Extreme 2-bit k-quant super-block compression with 4-bit group scales. Highly effective on fine-grained sparse MoE architectures (Qwen3-Coder-Next, Qwen3-30B-A3B), but exhibits sharp multi-step math degradation on dense architectures (see Section 4).

Target: 12–16 GB VRAM · Edge & MoE Offload
02 · Empirical Holdout Matrix

Cross-Model GGUF Accuracy & Compression Matrix

Every row below is rendered directly from our checked-in evaluation catalog (N=200 deterministic holdout questions per benchmark, seed 42, greedy argmax decoding). Filter by precision rung (Q4_K_M, Q8_0, Q2_K), architecture type (MoE vs Dense), or search by model family.

Precision:
Showing 32 evaluated configurations across 8 certified model releases (N=200 per benchmark, seed 42, greedy decoding).Zero-fabrication policy · Sourced from checked-in catalog
Qwen3.8-27B GGUF

58.0 GB · 1.00x

FP16 Base

ARC-MC

94.5%

GSM8K

59.0%

MATH-500

23.0%

Qwen3.8-27B GGUF

29.05 GB · 0.50x (-50%)

Q8_0

ARC-MC

94.5%

0.0 pp

GSM8K

63.5%

+4.5 pp

MATH-500

22.0%

-1.0 pp

Qwen3.8-27B GGUF

16.81 GB · 0.29x (-71%)

Q4_K_M

ARC-MC

95.0%

+0.5 pp

GSM8K

33.5%

-25.5 pp

MATH-500

17.5%

-5.5 pp

Qwen3.8-27B GGUF

10.86 GB · 0.19x (-81%)

Q2_K

ARC-MC

93.0%

-1.5 pp

GSM8K

86.0%

+27.0 pp

MATH-500

10.0%

-13.0 pp

Qwen3-30B-A3B GGUF

48.0 GB · 1.00x

FP16 Base

ARC-MC

93.5%

GSM8K

92.5%

MATH-500

49.5%

Qwen3-30B-A3B GGUF

24.77 GB · 0.52x (-48%)

Q8_0

ARC-MC

93.5%

0.0 pp

GSM8K

94.0%

+1.5 pp

MATH-500

48.5%

-1.0 pp

Qwen3-30B-A3B GGUF

14.16 GB · 0.29x (-71%)

Q4_K_M

ARC-MC

93.0%

-0.5 pp

GSM8K

94.0%

+1.5 pp

MATH-500

48.0%

-1.5 pp

Qwen3-30B-A3B GGUF

8.62 GB · 0.18x (-82%)

Q2_K

ARC-MC

86.5%

-7.0 pp

GSM8K

87.5%

-5.0 pp

MATH-500

38.5%

-11.0 pp

Gemma 4 26B-A4B MoE GGUF

50.5 GB · 1.00x

FP16 Base

ARC-MC

92.0%

GSM8K

73.0%

MATH-500

36.5%

Gemma 4 26B-A4B MoE GGUF

26.86 GB · 0.53x (-47%)

Q8_0

ARC-MC

92.0%

0.0 pp

GSM8K

70.5%

-2.5 pp

MATH-500

32.5%

-4.0 pp

Gemma 4 26B-A4B MoE GGUF

16.80 GB · 0.33x (-67%)

Q4_K_M

ARC-MC

92.5%

+0.5 pp

GSM8K

66.0%

-7.0 pp

MATH-500

23.0%

-13.5 pp

Gemma 4 26B-A4B MoE GGUF

10.58 GB · 0.21x (-79%)

Q2_K

ARC-MC

82.0%

-10.0 pp

GSM8K

4.5%

-68.5 pp

MATH-500

7.5%

-29.0 pp

Gemma 4 31B Dense GGUF

61.4 GB · 1.00x

FP16 Base

ARC-MC

85.0%

GSM8K

86.0%

MATH-500

36.5%

Gemma 4 31B Dense GGUF

32.64 GB · 0.53x (-47%)

Q8_0

ARC-MC

95.0%

+10.0 pp

GSM8K

84.5%

-1.5 pp

MATH-500

37.0%

+0.5 pp

Gemma 4 31B Dense GGUF

18.69 GB · 0.30x (-70%)

Q4_K_M

ARC-MC

94.5%

+9.5 pp

GSM8K

87.5%

+1.5 pp

MATH-500

35.5%

-1.0 pp

Gemma 4 31B Dense GGUF

11.92 GB · 0.19x (-81%)

Q2_K

ARC-MC

91.0%

+6.0 pp

GSM8K

2.0%

-84.0 pp

MATH-500

6.0%

-30.5 pp

Muse Glimmer 30B GGUF

59.55 GB · 1.00x

FP16 Base

ARC-MC

95.5%

GSM8K

95.5%

MATH-500

65.0%

Muse Glimmer 30B GGUF

29.61 GB · 0.50x (-50%)

Q8_0

ARC-MC

95.5%

0.0 pp

GSM8K

95.5%

0.0 pp

MATH-500

64.0%

-1.0 pp

Muse Glimmer 30B GGUF

16.94 GB · 0.28x (-72%)

Q4_K_M

ARC-MC

96.0%

+0.5 pp

GSM8K

93.0%

-2.5 pp

MATH-500

62.0%

-3.0 pp

Muse Glimmer 30B GGUF

10.69 GB · 0.18x (-82%)

Q2_K

ARC-MC

92.0%

-3.5 pp

GSM8K

74.0%

-21.5 pp

MATH-500

32.0%

-33.0 pp

Granite 4.2 30B GGUF

58.56 GB · 1.00x

BF16 Base

ARC-MC

92.0%

GSM8K

94.0%

MATH-500

54.5%

Granite 4.2 30B GGUF

31.11 GB · 0.53x (-47%)

Q8_0

ARC-MC

92.0%

0.0 pp

GSM8K

94.0%

0.0 pp

MATH-500

54.5%

0.0 pp

Granite 4.2 30B GGUF

17.72 GB · 0.30x (-70%)

Q4_K_M

ARC-MC

91.0%

-1.0 pp

GSM8K

92.5%

-1.5 pp

MATH-500

54.5%

0.0 pp

Granite 4.2 30B GGUF

10.86 GB · 0.19x (-81%)

Q2_K

ARC-MC

86.5%

-5.5 pp

GSM8K

68.0%

-26.0 pp

MATH-500

38.0%

-16.5 pp

LFM2.5-24B-A2B MoE GGUF

47.96 GB · 1.00x

BF16 Base

ARC-MC

88.0%

GSM8K

78.5%

MATH-500

58.5%

LFM2.5-24B-A2B MoE GGUF

25.49 GB · 0.53x (-47%)

Q8_0

ARC-MC

88.0%

0.0 pp

GSM8K

77.0%

-1.5 pp

MATH-500

54.5%

-4.0 pp

LFM2.5-24B-A2B MoE GGUF

14.53 GB · 0.30x (-70%)

Q4_K_M

ARC-MC

89.0%

+1.0 pp

GSM8K

78.5%

0.0 pp

MATH-500

50.5%

-8.0 pp

LFM2.5-24B-A2B MoE GGUF

8.81 GB · 0.18x (-82%)

Q2_K

ARC-MC

70.5%

-17.5 pp

GSM8K

54.5%

-24.0 pp

MATH-500

37.5%

-21.0 pp

Qwen3-Coder-Next MoE GGUF

159.46 GB · 1.00x

BF16 Base

ARC-MC

96.5%

GSM8K

95.5%

MATH-500

61.5%

Qwen3-Coder-Next MoE GGUF

84.81 GB · 0.53x (-47%)

Q8_0

ARC-MC

96.5%

0.0 pp

GSM8K

95.5%

0.0 pp

MATH-500

62.5%

+1.0 pp

Qwen3-Coder-Next MoE GGUF

48.49 GB · 0.30x (-70%)

Q4_K_M

ARC-MC

97.0%

+0.5 pp

GSM8K

95.5%

0.0 pp

MATH-500

61.0%

-0.5 pp

Qwen3-Coder-Next MoE GGUF

29.27 GB · 0.18x (-82%)

Q2_K

ARC-MC

96.5%

0.0 pp

GSM8K

91.5%

-4.0 pp

MATH-500

48.0%

-13.5 pp

03 · Methodology & Discordance

Why Aggregate Accuracy & Perplexity Are Not Enough: Item-Level McNemar Flips

As highlighted in the research literature “Accuracy is Not All You Need” (arXiv:2407.09141) and “Which Quantization Should I Use?” (arXiv:2601.14277), evaluating quantized LLMs solely by aggregate accuracy or WikiText perplexity is fundamentally flawed:

The Zero-Net-Delta Illusion (Error Cancellation)

When weights are quantized, some prompts that the FP16/BF16 base model answered correctly flip to incorrect (Flips Broke: Base ✓ → Quant ✗), while other prompts near the decision boundary flip from incorrect to correct (Flips Fixed: Base ✗ → Quant ✓). In aggregate metrics, these opposite flips cancel out!

Example: Granite 4.2 30B (Q4_K_M) scores 54.5% (109/200) on MATH-500 vs 54.5% (109/200) BF16 Base — a 0.0 pp net delta. Yet McNemar tracking reveals 16 broke and 16 fixed (32/200 items = 16.0% behavioral discordance).

Paired Contingency & Exact Identity Verification

Instead of independent averages, Omega Factory evaluates every GGUF rung on the exact same paired items i ∈ {1..200} under deterministic greedy decoding (seed 42) and enforces two algebraic invariants in CI:

1. QuantCorrect − BaseCorrect = FlipsFixed − FlipsBroke
2. NetDeltaPp = ((FlipsFixed − FlipsBroke) / 200) × 100
3. DiscordanceRate = (FlipsBroke + FlipsFixed) / 200
Quant Rung:
Task:
Showing 72 of 72 paired McNemar contingency rows across 8 certified models.Paired N=200 holdout · Exact identity verified
Qwen3.8-27B GGUF
Q8_0·ARC-MC
0.0 pp
Quant vs FP16189/200 vs 189
Flips Broke0
Flips Fixed0 (0/200 · 0%)
Qwen3.8-27B GGUF
Q8_0·GSM8K
+4.5 pp
Quant vs FP16127/200 vs 118
Flips Broke1
Flips Fixed10 (11/200 · 6%)
Qwen3.8-27B GGUF
Q8_0·MATH-500
-1.0 pp
Quant vs FP1644/200 vs 46
Flips Broke10
Flips Fixed8 (18/200 · 9%)
Qwen3.8-27B GGUF
Q4_K_M·ARC-MC
+0.5 pp
Quant vs FP16190/200 vs 189
Flips Broke0
Flips Fixed1 (1/200 · 1%)
Qwen3.8-27B GGUF
Q4_K_M·GSM8K
-25.5 pp
Quant vs FP1667/200 vs 118
Flips Broke61
Flips Fixed10 (71/200 · 36%)
Qwen3.8-27B GGUF
Q4_K_M·MATH-500
-5.5 pp
Quant vs FP1635/200 vs 46
Flips Broke21
Flips Fixed10 (31/200 · 16%)
Qwen3.8-27B GGUF
Q2_K·ARC-MC
-1.5 pp
Quant vs FP16186/200 vs 189
Flips Broke7
Flips Fixed4 (11/200 · 6%)
Qwen3.8-27B GGUF
Q2_K·GSM8K
+27.0 pp
Quant vs FP16172/200 vs 118
Flips Broke15
Flips Fixed69 (84/200 · 42%)
Qwen3.8-27B GGUF
Q2_K·MATH-500
-13.0 pp
Quant vs FP1620/200 vs 46
Flips Broke39
Flips Fixed13 (52/200 · 26%)
Qwen3-30B-A3B GGUF
Q8_0·ARC-MC
0.0 pp
Quant vs FP16187/200 vs 187
Flips Broke1
Flips Fixed1 (2/200 · 1%)
+1.5 pp
Quant vs FP16188/200 vs 185
Flips Broke1
Flips Fixed4 (5/200 · 3%)
Qwen3-30B-A3B GGUF
Q8_0·MATH-500
-1.0 pp
Quant vs FP1697/200 vs 99
Flips Broke4
Flips Fixed2 (6/200 · 3%)
Qwen3-30B-A3B GGUF
Q4_K_M·ARC-MC
-0.5 pp
Quant vs FP16186/200 vs 187
Flips Broke2
Flips Fixed1 (3/200 · 2%)
Qwen3-30B-A3B GGUF
Q4_K_M·GSM8K
+1.5 pp
Quant vs FP16188/200 vs 185
Flips Broke2
Flips Fixed5 (7/200 · 4%)
Qwen3-30B-A3B GGUF
Q4_K_M·MATH-500
-1.5 pp
Quant vs FP1696/200 vs 99
Flips Broke6
Flips Fixed3 (9/200 · 5%)
Qwen3-30B-A3B GGUF
Q2_K·ARC-MC
-7.0 pp
Quant vs FP16173/200 vs 187
Flips Broke14
Flips Fixed0 (14/200 · 7%)
-5.0 pp
Quant vs FP16175/200 vs 185
Flips Broke16
Flips Fixed6 (22/200 · 11%)
Qwen3-30B-A3B GGUF
Q2_K·MATH-500
-11.0 pp
Quant vs FP1677/200 vs 99
Flips Broke24
Flips Fixed2 (26/200 · 13%)
0.0 pp
Quant vs FP16184/200 vs 184
Flips Broke0
Flips Fixed0 (0/200 · 0%)
-2.5 pp
Quant vs FP16141/200 vs 146
Flips Broke8
Flips Fixed3 (11/200 · 6%)
-4.0 pp
Quant vs FP1665/200 vs 73
Flips Broke12
Flips Fixed4 (16/200 · 8%)
+0.5 pp
Quant vs FP16185/200 vs 184
Flips Broke4
Flips Fixed5 (9/200 · 5%)
-7.0 pp
Quant vs FP16132/200 vs 146
Flips Broke20
Flips Fixed6 (26/200 · 13%)
Gemma 4 26B-A4B MoE GGUF
Q4_K_M·MATH-500
-13.5 pp
Quant vs FP1646/200 vs 73
Flips Broke31
Flips Fixed4 (35/200 · 18%)
-10.0 pp
Quant vs FP16164/200 vs 184
Flips Broke24
Flips Fixed4 (28/200 · 14%)
-68.5 pp
Quant vs FP169/200 vs 146
Flips Broke138
Flips Fixed1 (139/200 · 70%)
-29.0 pp
Quant vs FP1615/200 vs 73
Flips Broke60
Flips Fixed2 (62/200 · 31%)
+10.0 pp
Quant vs FP16190/200 vs 170
Flips Broke0
Flips Fixed20 (20/200 · 10%)
-1.5 pp
Quant vs FP16169/200 vs 172
Flips Broke4
Flips Fixed1 (5/200 · 3%)
+0.5 pp
Quant vs FP1674/200 vs 73
Flips Broke0
Flips Fixed1 (1/200 · 1%)
+9.5 pp
Quant vs FP16189/200 vs 170
Flips Broke0
Flips Fixed19 (19/200 · 10%)
+1.5 pp
Quant vs FP16175/200 vs 172
Flips Broke1
Flips Fixed4 (5/200 · 3%)
Gemma 4 31B Dense GGUF
Q4_K_M·MATH-500
-1.0 pp
Quant vs FP1671/200 vs 73
Flips Broke3
Flips Fixed1 (4/200 · 2%)
+6.0 pp
Quant vs FP16182/200 vs 170
Flips Broke4
Flips Fixed16 (20/200 · 10%)
-84.0 pp
Quant vs FP164/200 vs 172
Flips Broke168
Flips Fixed0 (168/200 · 84%)
-30.5 pp
Quant vs FP1612/200 vs 73
Flips Broke61
Flips Fixed0 (61/200 · 31%)
0.0 pp
Quant vs FP16191/200 vs 191
Flips Broke0
Flips Fixed0 (0/200 · 0%)
0.0 pp
Quant vs FP16191/200 vs 191
Flips Broke0
Flips Fixed0 (0/200 · 0%)
Muse Glimmer 30B GGUF
Q8_0·MATH-500
-1.0 pp
Quant vs FP16128/200 vs 130
Flips Broke7
Flips Fixed5 (12/200 · 6%)
Muse Glimmer 30B GGUF
Q4_K_M·ARC-MC
+0.5 pp
Quant vs FP16192/200 vs 191
Flips Broke1
Flips Fixed2 (3/200 · 2%)
-2.5 pp
Quant vs FP16186/200 vs 191
Flips Broke6
Flips Fixed1 (7/200 · 4%)
Muse Glimmer 30B GGUF
Q4_K_M·MATH-500
-3.0 pp
Quant vs FP16124/200 vs 130
Flips Broke16
Flips Fixed10 (26/200 · 13%)
-3.5 pp
Quant vs FP16184/200 vs 191
Flips Broke8
Flips Fixed1 (9/200 · 5%)
-21.5 pp
Quant vs FP16148/200 vs 191
Flips Broke44
Flips Fixed1 (45/200 · 23%)
Muse Glimmer 30B GGUF
Q2_K·MATH-500
-33.0 pp
Quant vs FP1664/200 vs 130
Flips Broke70
Flips Fixed4 (74/200 · 37%)
0.0 pp
Quant vs BF16184/200 vs 184
Flips Broke0
Flips Fixed0 (0/200 · 0%)
0.0 pp
Quant vs BF16188/200 vs 188
Flips Broke2
Flips Fixed2 (4/200 · 2%)
Granite 4.2 30B GGUF
Q8_0·MATH-500
0.0 pp
Quant vs BF16109/200 vs 109
Flips Broke9
Flips Fixed9 (18/200 · 9%)
Granite 4.2 30B GGUF
Q4_K_M·ARC-MC
-1.0 pp
Quant vs BF16182/200 vs 184
Flips Broke3
Flips Fixed1 (4/200 · 2%)
-1.5 pp
Quant vs BF16185/200 vs 188
Flips Broke7
Flips Fixed4 (11/200 · 6%)
Granite 4.2 30B GGUF
Q4_K_M·MATH-500
0.0 pp
Quant vs BF16109/200 vs 109
Flips Broke16
Flips Fixed16 (32/200 · 16%)
-5.5 pp
Quant vs BF16173/200 vs 184
Flips Broke16
Flips Fixed5 (21/200 · 11%)
-26.0 pp
Quant vs BF16136/200 vs 188
Flips Broke57
Flips Fixed5 (62/200 · 31%)
Granite 4.2 30B GGUF
Q2_K·MATH-500
-16.5 pp
Quant vs BF1676/200 vs 109
Flips Broke43
Flips Fixed10 (53/200 · 27%)
0.0 pp
Quant vs BF16176/200 vs 176
Flips Broke3
Flips Fixed3 (6/200 · 3%)
-1.5 pp
Quant vs BF16154/200 vs 157
Flips Broke11
Flips Fixed8 (19/200 · 10%)
-4.0 pp
Quant vs BF16109/200 vs 117
Flips Broke17
Flips Fixed9 (26/200 · 13%)
+1.0 pp
Quant vs BF16178/200 vs 176
Flips Broke4
Flips Fixed6 (10/200 · 5%)
0.0 pp
Quant vs BF16157/200 vs 157
Flips Broke15
Flips Fixed15 (30/200 · 15%)
LFM2.5-24B-A2B MoE GGUF
Q4_K_M·MATH-500
-8.0 pp
Quant vs BF16101/200 vs 117
Flips Broke28
Flips Fixed12 (40/200 · 20%)
-17.5 pp
Quant vs BF16141/200 vs 176
Flips Broke40
Flips Fixed5 (45/200 · 23%)
-24.0 pp
Quant vs BF16109/200 vs 157
Flips Broke57
Flips Fixed9 (66/200 · 33%)
-21.0 pp
Quant vs BF1675/200 vs 117
Flips Broke48
Flips Fixed6 (54/200 · 27%)
0.0 pp
Quant vs BF16193/200 vs 193
Flips Broke0
Flips Fixed0 (0/200 · 0%)
0.0 pp
Quant vs BF16191/200 vs 191
Flips Broke3
Flips Fixed3 (6/200 · 3%)
+1.0 pp
Quant vs BF16125/200 vs 123
Flips Broke10
Flips Fixed12 (22/200 · 11%)
+0.5 pp
Quant vs BF16194/200 vs 193
Flips Broke0
Flips Fixed1 (1/200 · 1%)
0.0 pp
Quant vs BF16191/200 vs 191
Flips Broke2
Flips Fixed2 (4/200 · 2%)
-0.5 pp
Quant vs BF16122/200 vs 123
Flips Broke15
Flips Fixed14 (29/200 · 14%)
0.0 pp
Quant vs BF16193/200 vs 193
Flips Broke1
Flips Fixed1 (2/200 · 1%)
-4.0 pp
Quant vs BF16183/200 vs 191
Flips Broke11
Flips Fixed3 (14/200 · 7%)
-13.5 pp
Quant vs BF1696/200 vs 123
Flips Broke36
Flips Fixed9 (45/200 · 23%)
04 · Empirical Failure Mode Analysis

When 2-Bit (Q2_K) Breaks Down: Agentic, Coding & Multi-Step Math Workloads

Across our 8 certified model ladders, comparing ARC-MC (single-step multiple-choice science reasoning) against GSM8K and MATH-500 (multi-step chain-of-thought generation) exposes three critical engineering rules for production deployment:

1. Multiple-Choice Accuracy (ARC-MC) Masks Autoregressive Collapse in Dense 2-Bit Models

Critical Production Warning

Single-token multiple-choice evaluation only requires the correct option token (A/B/C/D) to remain the argmax over 4 logits. Multi-step math and agentic tool-calling, however, require generating hundreds of coherent reasoning tokens without a single syntax or arithmetic derailment.

Gemma 4 31B Dense (Q2_K · 11.92 GB)

  • • ARC-MC: 91.0% (+6.0 pp vs FP16) — looks healthy!
  • • GSM8K: 2.0% (-84.0 pp · 168/200 broke)
  • • MATH-500: 6.0% (-30.5 pp · 61/200 broke)

Gemma 4 26B-A4B MoE (Q2_K · 10.58 GB)

  • • ARC-MC: 82.0% (-10.0 pp vs FP16)
  • • GSM8K: 4.5% (-68.5 pp · 138/200 broke)
  • • MATH-500: 7.5% (-29.0 pp · 60/200 broke)

2. High-Granularity Sparse MoE Architectures Survive Extreme Compression Significantly Better

MoE Structural Advantage

When total capacity is distributed across many fine-grained routed experts (such as Qwen3-Coder-Next with 512 routed + 1 shared experts and only 3.14B active parameters per token, or Qwen3-30B-A3B with 3.3B active parameters), individual expert quantization noise averages out across top-k routing:

Qwen3-Coder-Next MoE (Q2_K · 29.27 GB · -82%)

  • • ARC-MC: 96.5% (0.0 pp vs 159.5 GB BF16)
  • • GSM8K: 91.5% (-4.0 pp · 183/200 correct)
  • • MATH-500: 48.0% (-13.5 pp · 96/200 correct)

Qwen3-30B-A3B MoE (Q2_K · 8.62 GB · -82%)

  • • ARC-MC: 86.5% (-7.0 pp vs 48.0 GB FP16)
  • • GSM8K: 87.5% (-5.0 pp · 175/200 correct)
  • • MATH-500: 38.5% (-11.0 pp · 77/200 correct)

3. Non-Monotonic Reasoning Trajectories in Vision-Language Dense Backbones (Qwen3.8-27B)

Quantization does not always degrade accuracy monotonically. In Qwen3.8-27B GGUF, greedy decoding on GSM8K shifts from 59.0% (FP16 Base) and 63.5% (Q8_0, +4.5 pp) down to 33.5% at Q4_K_M (-25.5 pp) due to verbosity/stopping-token boundary shifts under 4-bit block scales, while Q2_K (10.86 GB) reaches 86.0% (+27.0 pp, 69 fixed vs 15 broke) as quantization noise suppresses verbose over-thinking loops on grade-school arithmetic—even as harder competition math (MATH-500) follows the expected monotonic curve (23.0% FP16 → 22.0% Q8_0 → 17.5% Q4_K_M → 10.0% Q2_K). This proves why every single quantization rung must be empirically tested across multiple task complexity tiers rather than extrapolated from perplexity.

05 · Hardware Sizing & Memory Bandwidth

VRAM & Hardware Selection Guide (12 GB to 96 GB+)

Use this hardware matrix to match your local GPU VRAM or Apple Silicon Unified Memory capacity to the optimal certified Omega Factory GGUF checkpoint. Remember to reserve 2.0–4.5 GB of headroom above the raw GGUF disk size for the KV cache (-c 16384) and CUDA/Metal compute buffers.

12 – 16 GB VRAMMoE Q4_K_M / Q2_K

RTX 4070 / 4070 Ti / 4080, Mac M-Series 16–24 GB

32 – 48 GB VRAM30B Q8_0 or 80B MoE Q2_K / Q4_K_M

RTX 5090 (32 GB), Dual RTX 3090/4090, RTX 6000 Ada, Mac 64 GB

64 – 128 GB+ Memory80B MoE Q4_K_M / Q8_0

Dual/Quad Workstation GPUs, H100/A100 80GB, Mac Studio M-Ultra

06 · Local & Server Deployment

Running Certified GGUFs with Ollama, llama.cpp & Python

Select any certified model and quantization rung below to generate exact, copy-pasteable CLI commands for Ollama, llama-server (OpenAI-compatible API), llama-cli, MoE CPU expert offloading, and pinned-revision Python downloads.

2. Select Precision Rung
Qwen3-Coder-Next.q4_k_m.gguf·48.49 GB·64 GB+ unified memory / multi-GPU or CPU+GPU MoE offload (~70% compression vs BF16). Retains 100% GSM8K accuracy (95.5%), 97.0% ARC-MC (+0.5 pp), and 61.0% MATH-500 (-0.5 pp).
Ollama Direct Pull

Stream directly from the Omega-Factory Hugging Face repository:

ollama run hf.co/Omega-Factory/Qwen3-Coder-Next-GGUF:Q4_K_M
llama-server (OpenAI API)

Serve an OpenAI-compatible `/v1/chat/completions` endpoint with Flash Attention:

llama-server -hf Omega-Factory/Qwen3-Coder-Next-GGUF -m Qwen3-Coder-Next.q4_k_m.gguf --host 127.0.0.1 --port 8080 --ctx-size 16384 --n-gpu-layers 99 -fa on
llama-cli Interactive Chat

Run interactive terminal chat with full GPU layer offload (`-ngl 99`):

llama-cli -hf Omega-Factory/Qwen3-Coder-Next-GGUF -m Qwen3-Coder-Next.q4_k_m.gguf --ctx-size 16384 --n-gpu-layers 99 --temp 0.6 --min-p 0.01 -cnv
MoE Expert CPU Offload (-ot)

Keep attention & shared layers in VRAM while offloading routed FFN experts to system RAM:

llama-cli -hf Omega-Factory/Qwen3-Coder-Next-GGUF -m Qwen3-Coder-Next.q4_k_m.gguf --ctx-size 16384 --n-gpu-layers 99 -ot ".ffn_.*_exps.=CPU" --threads 16 --temp 0.6
Python Pinned-Revision Download (huggingface_hub + hf_transfer) & SHA-256 Checksum
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import hf_hub_download

path = hf_hub_download(
    repo_id="Omega-Factory/Qwen3-Coder-Next-GGUF",
    filename="Qwen3-Coder-Next.q4_k_m.gguf",
    revision="9a9112af264ac5baef1b73dea5fc89b5ca632fb3",
)
print("Downloaded:", path)

# Linux / macOS
echo "c6a4c85791cf23606278e3131a0d723d3f720c75d51f4624d99a33e7cff1d6db  Qwen3-Coder-Next.q4_k_m.gguf" | sha256sum --check

# Windows PowerShell 7+
(Get-FileHash .\Qwen3-Coder-Next.q4_k_m.gguf -Algorithm SHA256).Hash.ToLower() -eq "c6a4c85791cf23606278e3131a0d723d3f720c75d51f4624d99a33e7cff1d6db"
07 · Supply-Chain Security & Provenance

Cryptographic Provenance & SHA-256 Release Attestation

Unverified weight repositories on public hubs can silently mutate when files are re-uploaded. Every Omega Factory release enforces a strict fail-closed cryptographic contract (OMEGA_VERIFIED_PUBLIC_MODEL_V1) binding the recommended GGUF file's SHA-256 digest, the 40-character Hugging Face Git commit SHA, and the checked-in catalog SHA-256 (d853e89023b5b037…).

Qwen3.8-27B.q4_k_m.gguf (16.81 GB)
sha256: ba4aeb0670e6…49a6fadb
Qwen3-30B-A3B GGUF

qwen3-30b-a3b-gguf-v1

f8ddac64b7…
Qwen3-30B-A3B.q4_k_m.gguf (14.16 GB)
sha256: ebd160a8a0e4…a9bc37aa
gemma-4-26b-a4b.q4_k_m.gguf (16.80 GB)
sha256: b828701dba86…58a0ba50
gemma-4-31b.q4_k_m.gguf (18.69 GB)
sha256: 7584d5b09832…92b5e316
Muse Glimmer 30B GGUF

muse-glimmer-30b-gguf-v1

8f3d41daea…
muse-glimmer-30b.q4_k_m.gguf (16.94 GB)
sha256: 6d8112c565f5…93a4876a
Granite 4.2 30B GGUF

granite-4.2-30b-gguf-v1

a6100f66fa…
granite-4.2-30b.q4_k_m.gguf (17.72 GB)
sha256: cd7fc4faec09…8699482d
LFM2.5-24B-A2B.q4_k_m.gguf (14.53 GB)
sha256: fbdde70ee6cd…032b74a2
Qwen3-Coder-Next.q4_k_m.gguf (48.49 GB)
sha256: c6a4c85791cf…cff1d6db
08 · Scientific Citations & Technical Reports

Foundational Architecture & Quantization Literature

All public releases and evaluation protocols in the Omega Factory catalog are grounded in peer-reviewed and foundational arXiv literature, indexed via Schema.org ScholarlyArticle citations for Google Dataset Search.

Quantization & EvaluationarXiv:2407.09141

Accuracy is Not All You Need: Measuring Answer Flips in Compressed Large Language Models

Empirical Compression & Discordance Study

Demonstrates that aggregate benchmark accuracy masks high rates of item-level answer flips under quantization and layer pruning, establishing flip discordance and KL divergence as essential post-training evaluation metrics.

Core Model ArchitecturearXiv:2505.09388

Qwen3 Technical Report

Qwen Team, Alibaba Group

Introduces the Qwen3 foundation model suite across dense and Mixture-of-Experts (MoE) architectures, featuring unified thinking/non-thinking modes, dynamic reasoning budgets, and high-efficiency token routing.

Quantization & EvaluationarXiv:2601.14277

Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization

Empirical GGUF Research Study

Comprehensive empirical analysis of llama.cpp GGUF quantization schemes (Q8_0, Q4_K_M, Q2_K) across reasoning benchmarks, memory footprints, and CPU/GPU throughput.

Predecessor ArchitecturearXiv:2412.15115

Qwen2.5 Technical Report

Qwen Team, Alibaba Group

Foundational architecture, attention mechanisms, and pretraining scale underlying the predecessor Qwen2.5 dense models.

Core Model ArchitecturearXiv:2607.02770

Gemma 4 Technical Report

Gemma Team, Google DeepMind

Presents the Gemma 4 open-weight model family, introducing multimodal dense and MoE architectures, native thinking reasoning traces, and unified encoder-free audio and image processing.

Predecessor ArchitecturearXiv:2403.08295

Gemma: Open Models Based on Gemini Research and Technology

Gemma Team, Google DeepMind

Foundational architecture, safety evaluations, and training methodology derived from Google Gemini research and technology.

Core Model ArchitecturearXiv:2602.06036

DFlash: Block Diffusion for Flash Speculative Decoding

Jian Chen, Yesheng Liang, Zhijian Liu

Speculative decoding framework using lightweight block diffusion for parallel token drafting, achieving over 6x lossless acceleration and integrated into Muse Glimmer's native speculative architecture.

Predecessor ArchitecturearXiv:2504.13181

Perception Encoder: The best visual embeddings are not at the output of the network

Daniel Bolya, Po-Yao Huang, Peize Sun, et al.

State-of-the-art vision perception encoder (ViT-G/14, ~1.8B parameters) powering Muse Glimmer's multimodal interleaved image and text understanding.

Core Model ArchitecturearXiv:2405.04324

Granite Code Models: A Family of Open Foundation Models for Code Intelligence

IBM Research Granite Team

Introduces IBM Granite decoder-only foundation models trained on 116 programming languages with rigorous data governance, enterprise safety compliance, and tool calling.

Core Model ArchitecturearXiv:2511.23404

LFM2 Technical Report

Liquid AI

Introduces Liquid Foundation Models (LFM2), a hardware-aware hybrid architecture combining gated short convolutions with grouped-query attention and sparse mixture-of-experts routing for fast, low-memory on-device inference.

Predecessor ArchitecturearXiv:2412.06464

Gated Delta Networks: Improving Mamba2 with Delta Rule

Songlin Yang, Jan Kautz, Ali Hatamizadeh

Introduces Gated DeltaNet linear attention combining gating and delta-update memory rules, forming the 3:1 hybrid Gated DeltaNet + Gated Attention backbone of Qwen3-Coder-Next.

09 · Technical FAQ

Frequently Asked Questions on GGUF Quantization

Which GGUF quantization should I use: Q8_0, Q4_K_M, or Q2_K?

For 90% of local and production deployments, Q4_K_M is the recommended sweet spot: it reduces model disk and memory footprint by 67%–72% compared to FP16/BF16 while preserving near-baseline accuracy across science (ARC-MC) and mathematical reasoning (GSM8K, MATH-500). Choose Q8_0 (~47%–50% compression) when you have 32–96 GB of VRAM/Unified Memory and require virtually zero item-level discordance on complex coding or formal math. Reserve Q2_K (~79%–82% compression) strictly for memory-constrained edge hardware or high-granularity MoE architectures (such as Qwen3-Coder-Next or Qwen3-30B-A3B) where sparse routing preserves reasoning better than dense 2-bit quantization.

Why does Omega Factory report McNemar discordance flips instead of only perplexity or aggregate accuracy?

As demonstrated in 'Accuracy is Not All You Need' (arXiv:2407.09141), aggregate benchmark scores and corpus perplexity can hide substantial behavioral drift because questions that break under quantization (Base ✓ → Quant ✗) and questions that stochastically flip from wrong to right (Base ✗ → Quant ✓) cancel each other out in the mean. For example, Granite 4.2 30B Q4_K_M achieves 54.5% on MATH-500 — an exact 0.0 pp net delta against its BF16 baseline — yet item-level McNemar tracking over N=200 holdout prompts reveals 16 broken answers and 16 fixed answers (16.0% total discordance). Publishing both Flips Broke and Flips Fixed exposes the true trajectory stability of every GGUF rung.

Why shouldn't 2-bit (Q2_K) GGUFs be used uncritically for agentic tool calling and math?

Multiple-choice benchmarks like ARC-MC evaluate single-token log-probability ranking, which remains resilient even under extreme 2-bit quantization. However, multi-step autoregressive generation (such as GSM8K, MATH-500, and structured JSON tool calling) compounds quantization noise across hundreds of generated tokens. On dense models like Gemma 4 31B Dense, Q2_K retains 91.0% on ARC-MC (+6.0 pp) while collapsing on GSM8K from 86.0% to 2.0% (-84.0 pp, with 168/200 answers broken). Always inspect multi-step generative benchmarks (GSM8K and MATH-500) before deploying extreme low-bit quants in agentic pipelines.

How do I run Omega Factory GGUF models directly from Hugging Face with Ollama or llama.cpp?

All Omega Factory repositories use standard GGUF v3 metadata compatible with direct Hugging Face streaming. With Ollama, run `ollama run hf.co/Omega-Factory/<Repo-Name>:Q4_K_M`. With llama.cpp, run `llama-cli -hf Omega-Factory/<Repo-Name> -m <filename.gguf> -ngl 99 -c 16384` or launch an OpenAI-compatible HTTP server via `llama-server -hf Omega-Factory/<Repo-Name> -m <filename.gguf> --port 8080 -ngl 99`.

How can I run an 80B MoE model like Qwen3-Coder-Next on a single 24 GB GPU?

Qwen3-Coder-Next has 79.6B total parameters (48.49 GB in Q4_K_M) but activates only 3.14B parameters per token across its 512 routed + 1 shared experts. Using llama.cpp's tensor override flag (`-ot ".ffn_.*_exps.=CPU"`), you can keep all shared attention layers, router gates, and KV cache on your 24 GB GPU while streaming sparse FFN expert weights from dual-channel DDR5 system RAM—preserving Q4_K_M reasoning quality (97.0% ARC-MC, 95.5% GSM8K, 61.0% MATH-500) without dropping to ultra-low bit widths.

How do I verify the cryptographic integrity and provenance of a downloaded GGUF file?

Every Omega Factory release binds its artifacts to an immutable 40-character Hugging Face Git commit SHA and a 64-character SHA-256 digest in our checked-in catalog (`src/data/public-model-catalog.json`). Run `sha256sum <filename.gguf>` on Linux/macOS or `Get-FileHash <filename.gguf> -Algorithm SHA256` in PowerShell and compare the output against the pinned SHA-256 hash and `OMEGA_VERIFIED_PUBLIC_MODEL_V1` release marker on the model's detail page.

Production Model Optimization

Need a custom precision ladder or private model compression?

Explore all 8 public certified GGUF releases in our catalog, or partner with Omega Factory to compress, prune, and verify your proprietary checkpoints against your internal domain holdouts.

Docs bound to Catalog public_model_catalog_v1 · generated 2026-10-14T08:00:00Z · sha256 d853e89023b5b0378c96b28cf519639e2b0bb9b774d171f1fe1e0dbcc48b68f8