GCMP: LoRA quality with 55× fewer parameters

Qwen3.8-27B, Alpaca, 500 steps. Same evaluation code, prompts, and questions for every row. Greedy decoding. Significance from paired tests: exact McNemar for right/wrong tasks, bootstrap for losses.

What GCMP is

LoRA made fine-tuning practical by training small low-rank matrices on a frozen base. GCMP takes the same idea further. It trains far fewer parameters while reaching the same quality. On Qwen3.8-27B, a GCMP adapter has 2.9M trainable parameters, compared with ~159M for LoRA r=32.

Smaller adapters change the economics of fine-tuning. You can keep one adapter per customer, team, or task without a storage problem. You can load many of them onto one server and pick one per request. And because the base stays frozen, your data and adapters stay on your own hardware.

GCMP does not require quantization. It works on full-precision BF16/FP16 models and on 4-bit bases. In Training Factory, it is now the default adapter.

The base: CQ matches BF16, NF4 does not

An adapter can only build on what the base still knows. So before comparing adapters, we compared the 4-bit bases against full precision on nine downstream tasks.

• —CQ vs. BF16: no significant difference on any task.
• —NF4 vs. BF16: NF4 is 12.0 points lower on GSM8K (p = 3e-6).
• —CQ vs. NF4: CQ is 8.7 points higher on GSM8K (p = 0.0005) and 2.8 points higher on BoolQ (p = 0.02).

CQ is also smaller and closer to full precision on WikiText-2: an 18 GB checkpoint vs. 22.4 GB for NF4, mean KL divergence of 0.0282 vs. 0.0330, and 92.42% top-1 agreement vs. 91.95%. It needs no calibration data.

GCMP vs. LoRA r=32

To test GCMP on its own, we used the original BF16 Qwen3.8-27B with no quantization. Both adapters were trained with Unsloth using the same settings.

• —Benchmarks: no significant difference between GCMP and LoRA on any task (McNemar p ≥ 0.2 on every right/wrong task).
• —Answer-token loss: slightly lower for GCMP (−0.009, 95% CI [−0.010, −0.008]).
• —Validation loss: lower for GCMP at all 10 checkpoints, on both BF16 (1.2385 vs. 1.2434) and NF4 (1.2435 vs. 1.2484) bases.
• —Effect of fine-tuning: both adapters raise GSM8K by about 21 points and lower TruthfulQA by about 2.5 points, which is typical of Alpaca tuning.

Benchmarks for LoRA and GCMP on CQ and NF4 bases are still running. We will publish them when they finish.

Speed and memory

All runs use Alpaca, 500 steps, batch 2 × gradient accumulation 4, and sequence length 512.

GCMP uses slightly less peak memory than LoRA in every setup. It is not faster yet. Today it costs about 2× the step time of LoRA, and Unsloth is still faster than the Training Factory trainer. Kernel work to close that gap is in progress.

A correction to our earlier post

In our earlier QLoRA post, we compared training-log losses from the CQ trainer and Unsloth directly (0.84 vs. 1.25). The two trainers tokenize, mask, and average loss differently, so those numbers are not on the same scale. We no longer use that comparison. The same-formula losses and downstream benchmarks above replace it.

Limitations

• —Results so far cover one model (Qwen3.8-27B) and one dataset (Alpaca), using a quick benchmark subset that detects differences of about 5 points rather than 1.
• —Training runs on a single GPU. CQ serving supports pipeline parallelism but not tensor parallelism. vLLM serving of GCMP currently requires a CQ base.
• —MoE expert weights and non-CQ dense layers are not adapted.
• —Measure quality, speed, and memory on your own data before relying on these numbers.

Try it

Install Training Factory and download a CQ base, then train a GCMP adapter:

python training/train.py \
  --model ./models/Qwen3.8-27B-CQ-4bit \
  --data training/examples/train.jsonl \
  --output ./outputs/my-gcmp \
  --kernel-module cq.cq_kernels \
  --adapter gcmp

Serve several adapters from one CQ base and pick one per request with "model": "sql":

CQ_GCMP_ADAPTERS="support=./outputs/support,sql=./outputs/sql" \
vllm serve ./models/Qwen3.8-27B-CQ-4bit --quantization cq

Or train GCMP on a full-precision base with no quantization (80 GB GPU for 27B):

python training/train_unsloth_16bit_gcmp.py \
  --model Qwen/Qwen3.8-27B --dtype bfloat16 \
  --train-data training/examples/train.jsonl \
  --output ./outputs/my-gcmp-bf16

Guides: train and serve your first GCMP adapter, choose between BF16, CQ, and NF4, and serve many adapters from one base.