Research · 2025 – 2026
Silent ranking changes under dependency upgrades
A controlled study of what NumPy, SciPy, pandas and PyTorch upgrades do to recommender outputs: 201 cases, four mechanisms, no quality loss.
Upgrading NumPy from 2.0.2 to 2.1.3 under an unchanged ALS recommender — same data, seeds, initial factors and settings — changed the top-10 list for 8.5% of test users on one machine. Nothing failed. On two other processors the same upgrade produced identical models.
That example is the whole question: when a dependency upgrade completes and no alarm fires, did the recommender change, and did it get worse?
Design
Both sides of each upgrade run the same pipeline on frozen, checksummed inputs inside locked environments. A separately pinned evaluator hashes every checkpoint — data, model, scores, ranked list — and compares them. Eleven pipeline configurations on MovieLens 1M, with a MovieLens 20M scale check, three CPU types and a GPU extension.
Findings
| Outcome | Cases |
|---|---|
| Identical checkpoints on both sides | 164 |
| A checkpoint changed silently | 14 |
| …of which a top-10 list changed | 9 |
| Visible runtime failure | 16 |
| Not applicable | 7 |
| Quality regression or contract violation | 0 |
Every silent change traces to one of four mechanisms, each confirmed by intervention: NumPy 2.0’s unstable tie-breaking, OpenBLAS switching kernel family, a bundled OpenBLAS binary changing a matrix inverse, and a last-bit change in log1p. The size of a numerical difference does not predict whether a list changes.