Samrand Hassan
English

A different list is not a worse list

What happens to a recommender when you upgrade NumPy: 201 controlled cases, four mechanisms, and no measurable loss of quality.

Upgrading NumPy from 2.0.2 to 2.1.3 under an unchanged ALS recommender — same data, same seeds, same initial factors, same settings — changed the top-10 list for 8.5% of test users on my main machine. Nothing failed. Both environments built, imported and finished every run. On a second AMD processor and on an Intel processor, the same upgrade produced bit-identical models.

So did the recommender get worse? No. Offline quality stayed within 0.002 NDCG@10.

That gap — between the output changed and the output got worse — is what this study is about.

Four questions, kept apart

For every upgrade I asked four separate questions, because answering one does not answer the others:

  1. Did a recorded checkpoint change — the data, the model, the scores or the ranked list?
  2. Did any user’s top-10 list change?
  3. Did the output break a contract: recommending something already seen, an invalid ID, a non-finite score?
  4. Did offline ranking quality get measurably worse?

What 201 cases showed

Thirty-five upgrade transitions of NumPy, SciPy, pandas and PyTorch, applied to eleven pipeline configurations on MovieLens 1M, each with a same-environment control, frozen-model inference and paired-seed retraining.

OutcomeCases
Identical checkpoints on both sides164
A checkpoint changed silently14
…of which a top-10 list changed9
Visible runtime failure16
Not applicable7
Quality regression or contract violation0

All fourteen silent changes came from upgrades that involved NumPy. The upgrades that broke a pipeline did it loudly — removed aliases like np.NaN and np.in1d, and a pandas<3 cap.

Where the changes came from

Each silent change traced to one of four mechanisms, and each mechanism was confirmed by intervening on it:

  • Ties. NumPy 2.0’s argpartition and argsort reorder exactly tied scores. 0.3–0.4% of users got a different list.
  • Kernels. The OpenBLAS bundled in the NumPy wheel switched kernel family on one CPU. Forcing the new core type in the old environment reproduced the upgraded ALS model bit for bit.
  • A bundled binary. A newer OpenBLAS changed an EASE matrix inverse by about 1e-15. No list changed.
  • One bit. NumPy 2.0’s log1p differs in the last bit. BM25 weights changed; lists did not.

The surprise is that the size of a numerical difference tells you nothing about whether lists change. Exact ties change lists with identical scores; differences of 1e-13 to 1e-11 changed none.

What to check before you ship an upgrade

No real upgrade was harmful, so I tested detectors on synthetic faults injected into real outputs. Contract checks caught every contract fault, even with a hundred users. A 300-user panel chosen at the ranking boundary caught every quality fault but raised a few false alarms. Exact-output checks caught everything — and also alarmed on almost every harmless change.

The practical rule: measure checkpoint changes, list changes, contract failures and relevance loss separately, and do not treat the first two as the last.

The code, protocols and paper are on GitHub; the paper has its own page under Research.

Get in touch

Research collaboration, a talk, or a hard engineering problem — write to me.