insideout Get Pro

16 · Layers that resist quantization

Pro

Inference and Serving, lesson 16

Layers that resist quantization

Outliers and sensitive layers, and how to find them.

You can quantize a row of weights by hand with a scale ("Quantizing weights").

Part 1. One big number ruins the scale

Here is a new row: 0.12, −0.08, 0.05, 0.21, −0.15, and one weight of 8.0. Five ordinary weights and one that's forty times bigger than the next largest.

Your turn

You quantize this row to INT4 (−7 to 7) with one symmetric scale. What do the five small weights come back as?

The scale is 8.0 ÷ 7 ≈ 1.14. Every small weight is less than a fifth of a step, so each one rounds to 0. Five weights were erased so that one weight could be stored well.

Example 1.1

The same row, with and without the outlier

  1. INT8, with 8.0scale 0.063
    0.126−0.0630.0630.189−0.1268.0

    INT8 survives, barely. The small weights get one to three steps each, and 0.05 comes back as 0.063, 26% off.

  2. INT4, with 8.0scale 1.14
    0.00.00.00.00.08.0

    INT4 wipes them out. Only the outlier is left.

  3. INT4, without 8.0scale 0.03
    0.12−0.090.060.21−0.15

    Take the outlier away and the same 4 bits do fine. The bits were never the problem. The scale was.

One outlier against the scale, in Python
def quantize(row, qmax):
    scale = max(abs(x) for x in row) / qmax
    return [round(round(x / scale) * scale, 3) for x in row]

row = [0.12, -0.08, 0.05, 0.21, -0.15, 8.0]
print(quantize(row, 127))
print(quantize(row, 7))
print(quantize(row[:5], 7))
# [0.126, -0.063, 0.063, 0.189, -0.126, 8.0]
# [0.0, 0.0, 0.0, 0.0, 0.0, 8.0]
# [0.12, -0.09, 0.06, 0.21, -0.15]

Drag the outlier up and down and watch what happens to the error on every other weight.

Quantize a row with an outlierLive

Part 2. Outlier features

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Outlier features
  2. Not every layer is equal
See Pro plansI have Pro: sign in
Next lessonGPTQ and AWQTwo methods that quantize to four bits with little loss. Start lesson 17

Inference and Serving overviewSpotted a mistake? Tell us.