insideout Get Pro

17 · GPTQ and AWQ

Pro

Inference and Serving, lesson 17

GPTQ and AWQ

Two methods that quantize to four bits with little loss.

You know scales and groups from "Quantizing weights", and why outliers hurt from "Layers that resist quantization".

Part 1. Rounding each weight alone

Round-to-nearest looks at each weight on its own. But the model never uses a weight on its own. It uses a layer's output: weights times inputs, added up. What you want to keep close is that sum.

Your turn

Two weights, 0.26 and 0.26, each multiply an input of 1.0, so the true output is 0.52. The grid only has marks at 0.2 and 0.3. Which pair gives the closest output?

0.3 + 0.3 = 0.6, off by 0.08. 0.2 + 0.2 = 0.4, off by 0.12. 0.3 + 0.2 = 0.5, off by only 0.02. Rounding one up and one down lets the errors cancel. Round-to-nearest picks 0.3 and 0.3, because it judges each weight alone.

Example 1.1

Rounding with the error carried forward

  1. w₁ 0.26→0.3
    error +0.04

    Round the first weight. It's now 0.04 too big.

  2. w₂ 0.26−0.04=0.22

    Before rounding the next weight, take the error off it. Its input is the same size, so this cancels the damage.

  3. w₂ 0.22→0.2

    Now round. 0.22 goes to 0.2.

  4. 0.3+0.2=0.5
    true0.52

    The second weight on its own is worse (0.2 against 0.26), but the output is much closer.

Rounding with the error carried forward, in Python
w = [0.26, 0.27, 0.24, 0.36, 0.17, 0.26]
x = [1.0] * len(w)
step = 0.1
true = sum(a * b for a, b in zip(w, x))

nearest = [round(v / step) * step for v in w]

comp, carry = [], 0.0
for v in w:
    q = round((v + carry) / step) * step
    carry = v + carry - q
    comp.append(q)

for name, q in [("nearest", nearest), ("compensated", comp)]:
    out = sum(a * b for a, b in zip(q, x))
    print(name, [round(v, 1) for v in q], round(abs(out - true), 2))
# nearest [0.3, 0.3, 0.2, 0.4, 0.2, 0.3] 0.14
# compensated [0.3, 0.2, 0.3, 0.3, 0.2, 0.3] 0.04

Part 2. GPTQ: fix the rest as you go

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. GPTQ: fix the rest as you go
  2. AWQ: protect the weights that matter
  3. From paper to checkpoint
See Pro plansI have Pro: sign in
Next lessonPrefix cachingReuse the work for a shared system prompt or document. Start lesson 18

Inference and Serving overviewSpotted a mistake? Tell us.