Awq Quantization

AWQ (Activation-aware Weight Quantization)

4-bit quantization that preserves salient weights based on activation patterns, achieving 3x speedup with minimal accuracy loss.

When to use AWQ

Use AWQ when:

• Need 4-bit quantization with 1

}

### GEMV (single-token generation)

```python
quant_config = {
    "version": "GEMV"  # 20% faster for batch_size=1
}

Limitation: Only batch size 1, not good for large context.

Marlin (Ampere+ GPUs)

from transformers import AwqConfig, AutoModelForCausalLM

config = AwqConfig(
    bits=4,
    version="marlin"  # 2x faster on A100/H100
)

model = AutoModelForCausalLM.from_pretrained(
    "TheBloke/Mistral-7B-AWQ",
    quantization_config=config
)

Requirements: Compute Capability 8.0+ (A100, H100, RTX 40xx)

ExLlamaV2 (AMD compatible)

config = AwqConfig(
    bits=4,
    version="exllama"  # Faster prefill, AMD GPU support
)

HuggingFace Transformers integration

Direct loading

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "TheBloke/zephyr-7B-alpha-AWQ",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("TheBloke/zephyr-7B-alpha-AWQ")

Fused modules (recommended)

from transformers import AwqConfig, AutoModelForCausalLM

config = AwqConfig(
    bits=4,
    fuse_max_seq_len=512,  # Max sequence length for fusing
    do_fuse=True           # Enable fused attention/MLP
)

model = AutoModelForCausalLM.from_pretrained(
    "TheBloke/Mistral-7B-OpenOrca-AWQ",
    quantization_config=config
)

Note: Fused modules cannot combine with FlashAttention2.

vLLM integration

from vllm import LLM, SamplingParams

# vLLM auto-detects AWQ models
llm = LLM(
    model="TheBloke/Llama-2-7B-AWQ",
    quantization="awq",
    dtype="half"
)

sampling = SamplingParams(temperature=0.7, max_tokens=200)
outputs = llm.generate(["Explain AI"], sampling)

Performance benchmarks

Memory reduction

| Model | FP16 | AWQ 4-bit | Reduction | |-------|------|-----------|-----------| | Mistral 7B | 14 GB | 5.5 GB | 2.5x | | Llama 2-13B | 26 GB | 10 GB | 2.6x | | Llama 2-70B | 140 GB | 35 GB | 4x |

Inference speed (RTX 4090)

| Model | Prefill (tok/s) | Decode (tok/s) | Memory | |-------|-----------------|----------------|--------| | Mistral 7B GEMM | 3,897 | 114 | 5.55 GB | | TinyLlama 1B GEMV | 5,179 | 431 | 2.10 GB | | Llama 2-13B GEMM | 2,279 | 74 | 10.28 GB |

Accuracy (perplexity)

| Model | FP16 | AWQ 4-bit | Degradation | |-------|------|-----------|-------------| | Llama 3 8B | 8.20 | 8.48 | +3.4% | | Mistral 7B | 5.25 | 5.42 | +3.2% | | Qwen2 72B | 4.85 | 4.95 | +2.1% |

Custom calibration data

# Use custom dataset for domain-specific models
model.quantize(
    tokenizer,
    quant_config=quant_config,
    calib_data="wikitext",       # Or custom list of strings
    max_calib_samples=256,       # More samples = better accuracy
    max_calib_seq_len=512        # Sequence length
)

# Or provide your own samples
calib_samples = [
    "Your domain-specific text here...",
    "More examples from your use case...",
]
model.quantize(tokenizer, quant_config=quant_config, calib_data=calib_samples)

Multi-GPU deployment

model = AutoAWQForCausalLM.from_quantized(
    "TheBloke/Llama-2-70B-AWQ",
    device_map="auto",  # Auto-split across GPUs
    max_memory={0: "40GB", 1: "40GB"}
)

Supported models

35+ architectures including:

• Llama family: Llama 2/3, Code Llama, Mistral, Mixtral
• Qwen: Qwen, Qwen2, Qwen2.5-VL
• Others: Falcon, MPT, Phi, Yi, DeepSeek, Gemma
• Multimodal: LLaVA, LLaVA-Next, Qwen2-VL

Common issues

CUDA OOM during quantization:

# Reduce batch size
model.quantize(tokenizer, quant_config=quant_config, max_calib_samples=64)

Slow inference:

# Enable fused layers
model = AutoAWQForCausalLM.from_quantized(model_name, fuse_layers=True)

AMD GPU support:

# Use ExLlama backend
config = AwqConfig(bits=4, version="exllama")

Deprecation notice

AutoAWQ is officially deprecated. For new projects, consider:

• vLLM llm-compressor: https://github.com/vllm-project/llm-compressor
• MLX-LM: For Mac devices with Apple Silicon

Existing quantized models remain usable.

References

• Paper: AWQ: Activation-aware Weight Quantization (arXiv:2306.00978) - MLSys 2024 Best Paper
• GitHub: https://github.com/casper-hansen/AutoAWQ
• MIT Han Lab: https://github.com/mit-han-lab/llm-awq
• Models: https://huggingface.co/models?library=awq

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Orchestra-Research
• Source: Orchestra-Research/AI-Research-SKILLs
• License: MIT
• Homepage: http://orchestra-research.com

Install and usage instructions live in the source repository linked above.