insideout Get Pro

01 · Arrays, not loops

Free

Python for Market Data, lesson 1

Arrays, not loops

numpy: doing maths on a whole column at once.

Market data is long columns of numbers, and research code does its maths on whole columns at once with numpy. Every later lesson in this module works that way. You need basic Python (lists, loops and functions) and no finance.

By the end, you can:

  1. Turn an array of prices into daily returns without a loop.
  2. Count and pick the days that meet a condition, with a mask.
  3. Work on many stocks at once, and choose the axis to squash.
  4. Find a price series' maximum drawdown with a running maximum.

Part 1. Returns from a whole column

Turn an array of prices into daily returns without a loop.

A daily return is today's price over yesterday's, minus 1. With a list you'd loop: for each day, divide by the day before. numpy does every day in one line, once you line the prices up right.

Example 1.1

Five prices, four returns

p = np.array([100.0, 102.0, 101.0, 104.0, 103.0]) holds five closing prices, oldest first. Work out every daily return without a loop.

  1. Drop the first pricep[1:] → [102, 101, 104, 103]

    Every price but the first: each day that has a day before it.

  2. Drop the last pricep[:-1] → [100, 102, 101, 104]

    Every price but the last: the day before each of those.

  3. Divide, element by elementp[1:] / p[:-1] → [1.0200, 0.9902, 1.0297, 0.9904]

    The two arrays have the same length, so numpy divides the first by the first, the second by the second, and so on.

  4. Take 1 away- 1 → [0.0200, −0.0098, 0.0297, −0.0096]

    One number, taken from every element.

Answerr = p[1:] / p[:-1] - 1: four returns, +2.00%, −0.98%, +2.97% and −0.96%.

Stack the two slices and each price sits right over the one before it, so one division does every day at once. A Python loop does the same sums one by one, and pays a toll on each: fetch the number, check its type, divide, store. numpy pays that toll once per call, then loops in compiled code.

p100102101104103p[1:]102101104103p[:-1]100102101104firstdroppedlastdropped÷, then − 1+2.00%−0.98%+2.97%−0.96%r102 / 100 − 1
Figure 1p[1:] drops the first price and p[:-1] drops the last. Stacked, each price sits over the one before it, and one division gives every return.

Array, slice and vectorised

An array is numpy's block of numbers, all of one type, packed side by side. p[1:] is a slice: a stretch of the array, picked by position. Code that works on a whole array in one call is vectorised. The loop still runs, inside numpy's compiled code.

r = p[1:] / p[:-1] - 1

Two edges. n prices give n − 1 returns: the first day has no day before it. And numpy's lead grows with length. On a handful of numbers its fixed cost per call can make it no faster than a loop; on a million it's many times faster, as the timed code at the end shows.

Your turn

p = np.array([50.0, 55.0, 44.0, 44.0]). What does p[1:] / p[:-1] - 1 give?

55 / 50 − 1 = 0.1, 44 / 55 − 1 = −0.2 and 44 / 44 − 1 = 0. Four prices give three returns. [1.1, 0.8, 1.0] forgets the − 1, and [−0.0909, 0.25, 0.0] divides each day by the day after.

Part 2. Masks pick and count days

Count and pick the days that meet a condition, with a mask.

Many questions are about some of the days: how many were up, what the up days added, which days fell more than 2%. A comparison on the whole array answers them, again with no loop.

Example 2.1

Up days

r holds the four returns from part 1: [0.0200, −0.0098, 0.0297, −0.0096]. How many days were up, what share of days is that, and what did the up days add up to?

  1. Compare every day with 0up = r > 0 → [True, False, True, False]

    numpy compares each element with the single 0 and keeps the answers, one per day.

  2. Count the up daysup.sum() → 2

    True counts as 1 and False as 0.

  3. Take the shareup.mean() → 0.5

    The mean of True and False values is the share that are True.

  4. Add up only the up daysr[up].sum() → 0.0497

    r[up] keeps the elements where up is True: 0.0200 and 0.0297.

Answer2 up days out of 4, a share of 0.5, adding up to 0.0497, or 4.97%.

r > 0 works like the - 1 in part 1: numpy stretches the single number to the array's length, then goes element by element. What comes back is an array of True and False that lines up with r, day for day. That's why it can count the days, and pick them out.

Broadcasting and mask

Stretching a single number across a whole array is broadcasting. The array of True and False from a comparison is a mask. r[mask] keeps the elements where the mask is True, and mask.mean() is the share that are True.

Two conditions need & for and, | for or, and brackets round each one: r[(r > 0) & (r < 0.025)]. Python's own and raises a ValueError on arrays, and without the brackets Python reads the line in the wrong order and fails.

Also watch the flat days. r > 0 leaves them out and r >= 0 counts them, so say which you mean.

Your turn

r = np.array([0.02, -0.01, 0.03, 0.00]). What does (r >= 0).mean() return?

r >= 0 is [True, False, True, True]. Three of the four are True, and the mean of True and False is the share that are True: 0.75. (r >= 0).sum() would give the count, 3.

Part 3. Days down, stocks across

Work on many stocks at once, and choose the axis to squash.

Research uses many stocks at once. Put days down the rows and stocks across the columns, and every line so far works on all of them: P[1:] / P[:-1] - 1 gives every stock's returns. Then you usually want one number per stock, or one per day.

Example 3.1

Averages two ways

R holds four days of returns, in %, for GULL, KITE and WREN, one column each: GULL 1, 3, 2, 2; KITE −2, 0, 5, −3; WREN 4, −6, 2, 4. Find each stock's average return and each day's average.

  1. Read the shapeR.shape → (4, 3)

    Four rows of days, three columns of stocks. Five days of prices would give these four rows of returns.

  2. Squash the daysR.mean(axis=0) → [2, 0, 1]

    axis=0 runs down the rows, so each column becomes one number: one per stock.

  3. Squash the stocksR.mean(axis=1) → [1, −1, 3, 1]

    axis=1 runs across the columns: one number per day.

AnswerThe stocks average 2%, 0% and 1% a day. The days average 1%, −1%, 3% and 1%.

GULLKITEWRENaxis=1day 1+1−2+4+1day 2+30−6−1day 3+2+5+2+3day 4+2−3+4+1+20+1axis=0down the days: one per stock, (3,)across: oneper day, (4,)
Figure 2axis=0 squashes down the days and leaves one number per stock. axis=1 squashes across the stocks and leaves one per day.

Think of axis as the direction that disappears. Squash axis 0, the rows, and the shape (4, 3) loses its 4: you get (3,). Squash axis 1 and it loses its 3: (4,). The mean of each row is the return of a portfolio with equal money in each stock that day.

Shape and axis

An array's shape counts its elements in each direction: (4, 3) is 4 rows and 3 columns. The axis argument says which direction a sum, mean or standard deviation runs along. That direction disappears from the shape.

Leave out axis and numpy squashes everything into one number: R.mean() is 1, the average of all twelve cells. For a table of stocks that's rarely what you want, and it raises no error, so check the shape of what comes back.

Your turn

P holds 250 days of prices for 4 stocks, shape (250, 4), and R = P[1:] / P[:-1] - 1. What shape is R.std(axis=0)?

R has one row fewer than P: (249, 4). axis=0 squashes the days and leaves one volatility per stock: (4,). If you said (249,), you squashed the stocks instead, which is axis=1.

Part 4. The worst fall from a peak

Find a price series' maximum drawdown with a running maximum.

A common interview task: given an array of prices, find the biggest fall from a peak, without a loop. Say it in words first. On each day, how far is the price below the highest price so far? The biggest of those gaps is the answer.

Example 4.1

Eight days of prices

p = np.array([100, 108, 103, 111, 96, 99, 115, 104.0]). What's the biggest fall from a peak to a later low, and on which day does it bottom out?

  1. Find the running peak[100, 108, 108, 111, 111, 111, 115, 115]

    peak = np.maximum.accumulate(p) carries the highest price so far along the array, in one call.

  2. Measure each day against itdd = p / peak - 1 → [0, 0, −4.6%, 0, −13.5%, −10.8%, 0, −9.6%]

    How far each day sits below the peak before it.

  3. Take the lowestdd.min() → −13.5%

    dd.argmin() gives its position, 4: the day the price fell from 111 to 96.

AnswerA maximum drawdown of −13.5%, from 111 on day 3 to 96 on day 4.

running peak−13.5%100day 0108day 1103day 2111day 396day 499day 5115day 6104day 7
Figure 3The dashed line is the running peak. The biggest gap below it is on day 4: from 111 down to 96, or −13.5%.

Maximum drawdown

The biggest fall from a running peak to a later low, as a share of the peak. Risk reports quote it next to volatility: it's the loss of someone who bought at the worst moment.

(p / np.maximum.accumulate(p) - 1).min()

The trap is the highest price against the lowest. Here that's 96 / 115 − 1 = −16.5%, but the low, on day 4, came before the high, on day 6, so nobody ever lost that much. The running peak only looks back, which keeps the order right.

To check fast code, write the slow, obvious version too and compare. Two different ways of computing a thing rarely make the same mistake.

Max drawdown, checked with a loop
import numpy as np

p = np.array([100, 108, 103, 111, 96, 99, 115, 104.0])
peak = np.maximum.accumulate(p)     # the highest price so far
dd = p / peak - 1                   # how far below that peak
print(dd.round(3))
# [ 0.     0.    -0.046  0.    -0.135 -0.108  0.    -0.096]
print(dd.min().round(3), dd.argmin())
# -0.135 4

# The same answer the slow way, to check
worst, high = 0.0, p[0]
for x in p:
    high = max(high, x)
    worst = min(worst, x / high - 1)
print(round(worst, 3))
# -0.135

Your turn

p = np.array([50, 60, 45, 55, 70, 63.0]). What is its maximum drawdown?

The running peak is [50, 60, 60, 60, 70, 70], so the gaps are 0, 0, −25%, −8.3%, 0 and −10%. The lowest is 45 / 60 − 1 = −25%. The 70 came after the 45, so a fall from 70 to 45 never happened.

Where you'll meet it

In interviews

Turning a loop into array code is a common task in quant developer and research screens: returns, a running maximum, a drawdown, a count of up days. A frequent follow-up is a step that needs the one before, such as a position that depends on yesterday's. Making it fast, the last lesson in this module, covers those.

On the job

Research code runs over years of data for thousands of stocks. Twenty years of about 252 trading days for 3,000 stocks is about 15 million prices for each field, such as the close. On a laptop, a Python loop takes about a second for one pass over that, and the numpy line a few thousandths of a second. pandas, which the rest of this module uses, keeps its number columns as numpy arrays.

The two ways, timed
import time
import numpy as np

rng = np.random.default_rng(0)
p = 100 * np.cumprod(1 + rng.normal(0, 0.01, 1_000_000))

def returns_loop(p):
    out = []
    for i in range(1, len(p)):
        out.append(p[i] / p[i - 1] - 1)
    return np.array(out)

t0 = time.perf_counter()
a = returns_loop(p)
t1 = time.perf_counter()
b = p[1:] / p[:-1] - 1
t2 = time.perf_counter()

print(len(b), np.allclose(a, b))
# 999999 True
print("numpy at least 20x faster:", t1 - t0 > 20 * (t2 - t1))
# numpy at least 20x faster: True

Times change from machine to machine, so the code prints only what holds on any of them: the answers agree, and the gap is wide. In a Jupyter notebook, %timeit runs a line many times and reports the typical time.

Check yourself

Three questions from across the lesson. Try each one before you look back.

1

p holds a million prices. How does [np.log(x) for x in p] compare with np.log(p)?

The loop calls numpy a million times, and each call pays its fixed cost for one number. np.log(p) pays it once and loops in compiled code. On a laptop the gap is around a hundred times, for the same answers.

2

You want the returns between 0 and 1%. Which line works?

& combines two masks element by element, and the brackets make each comparison happen first. and raises a ValueError on arrays, the third line fails because & binds before >, and + on two masks acts as or, so it keeps every day.

3

An array p holds 251 daily closes. How long are p / np.maximum.accumulate(p) - 1 and p[1:] / p[:-1] - 1?

The drawdown compares every day with its running peak, the first day included, so it keeps all 251. A return needs a day before it, so the first day drops out: 250.

What to remember

  1. 1Returns from a whole column

    p[1:] / p[:-1] - 1 turns n prices into n − 1 returns with no loop: five prices give four returns.

  2. 2Masks pick and count days

    A comparison makes a mask: (r > 0).sum() counts the up days and (r > 0).mean() is their share, 0.5 for two in four.

  3. 3Days down, stocks across

    axis names the direction that disappears: on shape (4, 3), axis=0 leaves one number per stock, (3,), and axis=1 one per day, (4,).

  4. 4The worst fall from a peak

    Maximum drawdown is the lowest value of p / np.maximum.accumulate(p) - 1: −13.5% for a fall from 111 to 96.

Next lessonTables of pricesNext, the same arrays get labels: dates down the side and tickers across the top, in a pandas DataFrame. Start lesson 2

Python for Market Data overviewSpotted a mistake? Tell us.