ThousandML / Journal
Fine-tune on a quantized base. Just make sure the base is good.
The cheapest way to adapt an open model is to stop paying for full-precision weights you never update. Training adapters on top of an already-quantized model does exactly that, and it lowers the GPU requirement enough that enterprises can fine-tune in-house instead of sending private data to someone else's cluster. The catch: everything you train inherits the quality of that quantized base. That is why we built CQ.
The argument in three lines
- 1.Fine-tuning a quantized model is far cheaper than fine-tuning at full precision, which makes private, in-house fine-tuning practical.
- 2.It only works if the quantized base preserves the original model faithfully.
- 3.CQ is built for that: calibration-free, and grounded in rate–distortion theory.
Where fine-tuning cost actually goes
When teams adapt an open model to their domain, most of the GPU memory is not spent on what they are learning. It is spent holding the base model's weights. With parameter-efficient methods such as LoRA, the trainable adapter is a tiny fraction of the model, yet the frozen base still sits in memory at 16 bits per weight for every step of training.
That frozen base sets the floor on which GPU you need, how many you need, and how long you rent them. Shrink the base, and the floor drops with it.
Why train on an already-quantized model
If the base weights are frozen anyway, there is no reason to keep them at full precision. Storing them in a compressed low-bit format and training small adapters on top gives you three practical wins:
Smaller hardware
A 4-bit base needs roughly a quarter of the weight memory of FP16, so larger models fit on smaller, cheaper GPUs.
Tiny artifacts
You ship a small adapter per task instead of a full model copy per customer or domain.
No surprise at deploy
The model you train on is the quantized model you serve. You are not quantizing after training and hoping quality survives.
Cheaper fine-tuning makes private fine-tuning possible
For many enterprises, the hard limit on fine-tuning is privacy, not budget. Customer records, contracts, clinical notes, and source code often can't leave the company's network. Uploading them to a third-party fine-tuning service may be ruled out by policy, regulation, or contract.
Until now, the alternative was to fine-tune a full-precision model in-house, which can require large multi-GPU clusters that most teams don't have. A 4-bit CQ base uses roughly a quarter of the weight memory. That can make fine-tuning fit on GPUs a company already owns or rents in its own cloud account.
What stays inside your environment
- —Your training data, which never goes to an outside service
- —The adapters you train, which encode what the model learned from that data
- —The quantization step itself, because CQ needs no calibration data
Lower cost and private deployment come from the same change. A smaller base can turn fine-tuning from an outsourced project into something your team can run itself.
The base quality is everything
Here is the part that is easy to overlook. An adapter learns a small correction on top of the base. It is good at teaching the model a new format, vocabulary, or task. It is not designed to repair damage done to the base model's knowledge during compression.
If quantization has eroded reasoning, factual recall, or language ability, every fine-tune you run starts from that weaker point. You pay less for training, but you get a worse model, and the savings stop making sense.
Cheap fine-tuning is only worth it when the quantized base is close enough to the original that your adapter is learning your task, not compensating for compression.
We tested this directly. With the same QLoRA recipe on Qwen3.8-27B, an adapter trained on the CQ base reached 17.4% lower final validation loss than one trained on Unsloth's NF4 base. It was lower at every checkpoint. See the full benchmark.
So the real question is not “should I fine-tune a quantized model?” It is “how good is the quantized model I am fine-tuning?” Two properties of CQ answer that.
Property one: CQ is calibration-free
Most post-training quantization methods run a small set of representative inputs through the model and use the resulting statistics to decide how to compress each layer. That calibration set quietly shapes the quantized model: it preserves behavior the calibration data exercises and is less careful about everything else.
For inference on general chat, that can be fine. For fine-tuning, it is a structural problem. You are fine-tuning precisely because your domain is different — legal contracts, clinical notes, source code, a language other than English. A base quantized against generic calibration text may have been compressed with the least care in exactly the regions your domain depends on.
Calibration-based
Model + generic data → biased base → your domain fine-tune
CQ (calibration-free)
Model → domain-neutral base → your domain fine-tune
CQ quantizes directly from the weights. There is no calibration dataset and no calibration forward pass, so there is nothing to bias the base toward one domain over another. It also means no dataset to choose, version, or license, and data that cannot leave your environment never has to enter the quantization pipeline.
Property two: CQ is grounded in information theory
Removing calibration only helps if something principled takes its place. CQ — Coded Quantization — treats compressing model weights as a lossy source-coding problem: represent the weights with as few bits as possible while keeping the reconstruction as close to the original as possible.
Information theory gives a precise limit for that trade-off, the rate–distortion bound: the lowest distortion achievable at a given bit rate. CQ uses a proprietary coding-theoretic quantizer built on techniques designed to approach that bound, rather than on rounding heuristics tuned against a dataset.
Why this matters for fine-tuning
Because the objective is faithful reconstruction of the weights themselves, CQ aims to preserve the whole model at a given bit budget, not just the slice a calibration set happened to exercise. That is what you want underneath a fine-tune: a compressed base that stays as close as possible to the original everywhere, so your adapter can spend its capacity on your task.
Side by side
| Consideration | Full-precision fine-tuning | Training Factory on CQ |
|---|---|---|
| Base weights in memory | Full precision (FP16/BF16) | Compressed CQ weights |
| What is trained | All weights, or adapters on an FP16 base | Small adapters on a frozen CQ base |
| GPU needed for a given model | Larger, more expensive | Smaller, cheaper |
| Where training can run | Often outsourced to large cloud clusters | In-house, on GPUs you control |
| Train / deploy mismatch | Quantize after training, re-validate | Train on the model you deploy |
| Domain bias from quantization | Possible with calibration data | None — no calibration data used |
Full-precision training still has its place when budget is no object or when you need to update every weight. For most domain adaptation, a strong quantized base with adapters on top gets you there for a fraction of the cost.
Try it on your own data
Training Factory lets you take an open model, start from its CQ-quantized weights, and fine-tune adapters on your own dataset, on your own hardware. Your data stays on your machine. Run it against your workload and compare the result with what you would have paid at full precision.