A different list is not a worse list
What happens to a recommender when you upgrade NumPy: 201 controlled cases, four mechanisms, and no measurable loss of quality.
Upgrading NumPy from 2.0.2 to 2.1.3 under an unchanged ALS recommender — same data, same seeds, same initial factors, same settings — changed the top-10 list for 8.5% of test users on my main machine. Nothing failed. Both environments built, imported and finished every run. On a second AMD processor and on an Intel processor, the same upgrade produced bit-identical models.
So did the recommender get worse? No. Offline quality stayed within 0.002 NDCG@10.
That gap — between the output changed and the output got worse — is what this study is about.
Four questions, kept apart
For every upgrade I asked four separate questions, because answering one does not answer the others:
- Did a recorded checkpoint change — the data, the model, the scores or the ranked list?
- Did any user’s top-10 list change?
- Did the output break a contract: recommending something already seen, an invalid ID, a non-finite score?
- Did offline ranking quality get measurably worse?
What 201 cases showed
Thirty-five upgrade transitions of NumPy, SciPy, pandas and PyTorch, applied to eleven pipeline configurations on MovieLens 1M, each with a same-environment control, frozen-model inference and paired-seed retraining.
| Outcome | Cases |
|---|---|
| Identical checkpoints on both sides | 164 |
| A checkpoint changed silently | 14 |
| …of which a top-10 list changed | 9 |
| Visible runtime failure | 16 |
| Not applicable | 7 |
| Quality regression or contract violation | 0 |
All fourteen silent changes came from upgrades that involved NumPy. The upgrades that broke a pipeline did it loudly — removed aliases like np.NaN and np.in1d, and a pandas<3 cap.
Where the changes came from
Each silent change traced to one of four mechanisms, and each mechanism was confirmed by intervening on it:
- Ties. NumPy 2.0’s
argpartitionandargsortreorder exactly tied scores. 0.3–0.4% of users got a different list. - Kernels. The OpenBLAS bundled in the NumPy wheel switched kernel family on one CPU. Forcing the new core type in the old environment reproduced the upgraded ALS model bit for bit.
- A bundled binary. A newer OpenBLAS changed an EASE matrix inverse by about 1e-15. No list changed.
- One bit. NumPy 2.0’s
log1pdiffers in the last bit. BM25 weights changed; lists did not.
The surprise is that the size of a numerical difference tells you nothing about whether lists change. Exact ties change lists with identical scores; differences of 1e-13 to 1e-11 changed none.
What to check before you ship an upgrade
No real upgrade was harmful, so I tested detectors on synthetic faults injected into real outputs. Contract checks caught every contract fault, even with a hundred users. A 300-user panel chosen at the ranking boundary caught every quality fault but raised a few false alarms. Exact-output checks caught everything — and also alarmed on almost every harmless change.
The practical rule: measure checkpoint changes, list changes, contract failures and relevance loss separately, and do not treat the first two as the last.
The code, protocols and paper are on GitHub; the paper has its own page under Research.