Train and serve your first GCMP adapter

Before you start

GCMP is a parameter-efficient adapter that replaces LoRA. On Qwen3.8-27B it matched LoRA r=32 with 2.9M trainable parameters instead of ~159M, and the saved adapter is about 11 MB instead of ~320 MB. If you want the benchmarks first, read the GCMP release post.

You need:

• —Linux x86-64 or WSL2, with an NVIDIA GPU. A 48 GB card (for example an A40) is enough for a 27B model on a CQ base; peak memory was about 22 GiB in our runs.
• —Conda and Python 3.12. The native kernels ship compiled and must match your Python, CUDA, and PyTorch versions.
• —A ThousandML license check for the kernels. It is free during early access.

1. Install and download a CQ base

Clone the repository, run the installer, and download the 18 GB CQ checkpoint:

git clone https://github.com/thousandml/training-factory.git
cd training-factory
chmod +x install.sh
./install.sh

hf download thousandml/Qwen3.8-27B-CQ-4bit --local-dir ./models/Qwen3.8-27B-CQ-4bit

2. Prepare your data

Training data is JSONL, one example per line. Each line is either plain text or a chat-style messages list:

{"text": "Question: When does the warranty expire? Answer: Check the purchase date."}
{"messages": [{"role": "user", "content": "Summarize LoRA."}, {"role": "assistant", "content": "It trains small adapters on a frozen base."}]}

A small example dataset ships in training/examples/, so you can do a first run before preparing your own data. Keep a separate validation file: it lets you watch validation loss during training and catch overfitting early.

3. Train a GCMP adapter

GCMP is the default adapter, so --adapter gcmp is optional, but writing it out makes the command clear:

python training/train.py \
  --model ./models/Qwen3.8-27B-CQ-4bit \
  --data training/examples/train.jsonl \
  --eval-data training/examples/alpaca_val.jsonl --eval-steps 50 \
  --output ./outputs/my-gcmp \
  --kernel-module cq.cq_kernels \
  --adapter gcmp

The adapter is saved on its own as adapter_config.json and adapter_model.safetensors. Do not merge it into the CQ base: keeping it separate is what lets you serve many adapters from one base later.

Useful training options

Run python training/train.py --help for the full list. The --rank, --alpha, and --lora-dropout options only apply when --adapter lora.

4. Serve and call it

Point CQ_GCMP_ADAPTER at your output folder and start vLLM with the CQ quantization backend:

CQ_GCMP_ADAPTER=./outputs/my-gcmp \
vllm serve ./models/Qwen3.8-27B-CQ-4bit --quantization cq --max-model-len 8192

Then, in another terminal, once the server is ready:

curl localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "./models/Qwen3.8-27B-CQ-4bit",
  "messages": [{"role": "user", "content": "Explain fine-tuning in one sentence."}],
  "max_tokens": 128,
  "chat_template_kwargs": {"enable_thinking": false}
}'

Troubleshooting

First, check that the kernels import and CUDA is visible:

python -c "import torch, cq.cq_kernels; print('CUDA:', torch.cuda.is_available())"

• —Kernel import: the module is cq.cq_kernels, not cq_kernels.
• —Native binaries: they must match Python 3.12 and your CUDA and PyTorch versions.
• —License: set CQ_LICENSE_KEY / CQ_LICENSE_URL if your distribution needs one.
• —Windows line endings: if you see a bash\r error, run sed -i 's/\r$//' install.sh.