insideout Get Pro

12 · Arithmetic intensity

Pro

Inference and Serving, lesson 12

Arithmetic intensity

Operations per byte moved, and why decode is starved for bandwidth.

You know the roofline and its ridge point ("Reading a roofline chart"), and what the KV cache holds ("Caching keys and values").

Part 1. Counting one layer

Take one weight matrix from an 8B model: 4,096 rows by 4,096 columns, stored as 16-bit numbers. During decode, one token's vector of 4,096 numbers gets multiplied by it. Count what that costs on each side.

Example 1.1

One matrix, one token

  1. 4,096 × 4,096=16.8 million weights
    × 2 bytes=33.6 MB read

    Every weight has to come in from memory.

  2. 16.8 million weights×2 ops=33.6 million ops

    Each weight is used once: one multiply and one add.

  3. 33.6 million ops÷33.6 MB≈1 op per byte

    One operation for every byte read. On an H100, whose ridge is about 296, this sits far down the memory slope.

Arithmetic intensity

How many operations a piece of work does for each byte it moves in or out of memory. It's the number along the bottom of the roofline: left of the ridge point the reading sets the pace, right of it the arithmetic does.

Your turn

Store the same matrix as 8-bit numbers (1 byte each) instead of 16-bit. What happens to the arithmetic intensity of one decode token?

The operations don't change (33.6 million), but the bytes halve (16.8 MB). So intensity doubles and, since decode is memory-bound, the step gets close to twice as fast. That's the main reason people serve models in 8 or 4 bits, which "Number formats" picks up later in this module.

Part 2. Reuse raises it

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Reuse raises it
  2. Attention doesn't share
  3. Four ways to raise it
See Pro plansI have Pro: sign in
Next lessonContinuous batchingStatic, dynamic and continuous batching, and what each does to latency. Start lesson 13

Inference and Serving overviewSpotted a mistake? Tell us.