Skip to content
Home/Guides/Q4_K_M

GGUF quantization guide

Q4_K_M GGUF quantization explained

Q4_K_M is a four-bit-class weight quantization preset used in GGUF files for local LLMs. It reduces the precision used to store model weights, helping make model files smaller. GGUF is the file format; Q4_K_M describes the quantization. The VRAM needed to run the model also depends on context length, runtime settings, and GPU offload.

Quick answer

What is Q4_K_M in a GGUF file?

Q4_K_M describes a weight quantization preset in a GGUF file, not a required amount of VRAM. Quantization reduces the precision used to store model weights, which can make local model artifacts smaller. Current llama.cpp exposes Q4_K_M as a distinct quantization option and treats Q4_K as an alias for it. GGUF can store quantized tensors together with model metadata in a single model file.

What the label tells you

Separate quantization from runtime fit.

What Q4_K_M does tell you

  • It identifies a GGUF weight quantization preset.
  • It is part of the K-quant family used by llama.cpp.
  • It lets you distinguish one model artifact from other quantized variants of the same model.

What Q4_K_M does not tell you

  • Exact VRAM required at runtime.
  • Tokens per second on your GPU.
  • The maximum context your PC can use.
  • Whether full GPU offload is possible.

What do Q4, K, and M mean?

Q4: four-bit-class weights
The main weight representation uses four-bit values, plus scales and other data. It does not mean four bits for every tensor or 4 GB of memory.
K: the K-quant family
K-quants group weights into super-blocks with shared quantization information. The Q4_K block stores 256 weights with scales and minimum values; this block alone takes 4.5 bits per weight including that overhead.
M: the medium preset
M selects a tensor precision mix within the Q4 K-quant family. It is a preset label, not a model parameter count. The exact mixture can depend on model architecture and quantizer options.

GGUF is the container for tensors and metadata, including tokenizer information. It can hold higher-precision weights too; a .gguf extension alone does not specify quantization. See the GGUF specification and llama.cpp block definitions. The tensor selection implementation shows how medium presets retain selected weights at higher precision.

Reading an illustrative filename

ExampleModel-8B-Instruct-v1.0-Q4_K_M.gguf

This is an illustrative filename, not a downloadable model. ExampleModel names the model, 8B indicates its parameter class, Instruct describes instruction tuning, v1.0 is the version, Q4_K_M is the weight encoding preset, and .gguf is the file format. Check the repository and metadata rather than assuming every publisher follows the same naming convention.

Related labels

Q4_K_S vs Q4_K_M vs Q5_K_M

Q4_K_S

Small Q4 K-quant preset: prioritizes a smaller artifact. Consider it when the medium preset leaves too little memory headroom.

Llama-3-8B example: 4.37G

Q4_K_M

Medium Q4 K-quant preset: mixes tensor precisions, retaining selected weights at higher precision. A compromise between file size and quantization loss; Q4_K is its llama.cpp alias.

Llama-3-8B example: 4.58G

Q5_K_M

Medium Q5 K-quant preset: uses a higher-bit base representation and more storage. Consider it when memory allows, then test whether it helps your actual task.

Llama-3-8B example: 5.33G

These sizes compare the same Llama-3-8B model, exactly as reported (with the source's G notation) in the llama.cpp quantizer options. They are file-size examples, not VRAM requirements or AIPCFit measurements. More storage can reduce quantization loss, but does not guarantee better answers on your task. Quantizer versions and settings can change the resulting artifact.

Recorded examples

Q4_K_M models currently tracked by AIPCFit

These are specific recorded GGUF artifacts in the AIPCFit LM Studio dataset. File size is shown as source data only. It is not presented as required VRAM.

Qwen3 4B · Q4_K_M

Q4_K_M
Recorded file
2.5 GB
Recorded context
32,768

lmstudio-community/Qwen3-4B-GGUF

View recorded source

Qwen3 8B · Q4_K_M

Q4_K_M
Recorded file
5.03 GB
Recorded context
32,768

lmstudio-community/Qwen3-8B-GGUF

View recorded source

Qwen3 14B · Q4_K_M

Q4_K_M
Recorded file
9 GB
Recorded context
32,768

lmstudio-community/Qwen3-14B-GGUF

View recorded source

Gemma 3 4B · Q4_K_M

Q4_K_M
Recorded file
2.49 GB
Recorded context
32,768

lmstudio-community/gemma-3-4b-it-GGUF

View recorded source

Gemma 3 12B · Q4_K_M

Q4_K_M
Recorded file
7.3 GB
Recorded context
32,768

lmstudio-community/gemma-3-12b-it-GGUF

View recorded source

Gemma 3 27B · Q4_K_M

Q4_K_M
Recorded file
16.5 GB
Recorded context
32,768

lmstudio-community/gemma-3-27b-it-GGUF

View recorded source

Memory

Why GGUF file size is not the same as VRAM use

Loading a local LLM involves more than its model file. Runtime memory can also be affected by context length, KV cache, GPU offload, implementation details, and other load settings. AIPCFit therefore keeps recorded GGUF file size separate from its own planning heuristics.

Weights, the KV cache used to retain attention state, and compute buffers all compete for memory. Increasing context usually grows the KV cache; placing fewer layers on the GPU moves some weight storage and work to system RAM. A model that loads at short context may not fit fully on the GPU at a longer context.

KV-cache quantization is a separate runtime setting that compresses cached attention data. Choosing Q4_K_M weights does not choose the cache type. Support and quality trade-offs depend on the runtime and model; see the Ollama KV-cache documentation. Compare Qwen3 8B configurations with the 12GB VRAM guide before checking your complete PC.

Common questions

What is Q4_K_M?

Q4_K_M is a K-quantization preset used for GGUF model weights in the llama.cpp ecosystem. It describes how model weights are quantized; it is not a statement that a model needs 4 GB of VRAM.

Does Q4_K_M mean 4-bit?

The Q4 label refers to a four-bit-class K-quantization scheme, but the complete representation also includes scales and related quantization data. The label should not be converted directly into a final model file size or runtime memory number.

Is Q4_K_M file size the same as VRAM required?

No. A recorded GGUF file size is useful information, but runtime memory can also depend on context length, KV cache, GPU offload, runtime implementation, and other load settings.

Is Q4_K_M always the best quantization?

No universal quantization is best for every model, device, or workload. The useful choice depends on the specific model artifact, available hardware, and the trade-offs you accept.

Can LM Studio run Q4_K_M models?

AIPCFit's LM Studio catalog includes recorded Q4_K_M GGUF variants. Full practical fit still depends on the complete PC, including GPU VRAM, system RAM, CPU support, context, and offload settings.

Evidence and trust

File size is data. VRAM fit is a separate question.

AIPCFit keeps recorded model metadata, planning heuristics, application compatibility, and runtime uncertainty separate. The model examples above link directly to the repositories recorded in the catalog.

How AIPCFit evaluates local AI fit