Calibrating Reproduced Claims in Recommender Systems

2026-09-22

AI Summary

研究提出一种声明校准方法，用于明确复现研究支持的最强声明及其条件，解决推荐系统中复现研究结果不一致的问题。

Abstract

Reproduction studies can produce mixed outcomes. Reported values may differ while the ordering of the compared methods remains the same, a result may hold only under some experimental conditions, or a released implementation may fail to reproduce a result that the model can still reach. The terms repeatability, reproducibility, and replicability describe how a follow-up study relates to the original experiment, but not which parts of the original claim are supported by the new results. We introduce \emph{claim calibration} as a way of stating the strongest claim supported by a follow-up study, together with the conditions under which it holds and the parts that remain untested. We apply this perspective to five original--follow-up paper pairs from recommender-systems research. The cases show that agreement in numerical values, method rankings, statistical results, and overall conclusions does not always coincide, and that follow-up studies often support only part of the original claim. Based on these observations, we propose a Claim Evidence Profile for reporting the original claim, its scope, the reproduction target, the reported results, the calibrated claim, and the parts of the original claim that remain unresolved.

Problem

• Reproducibility
• Recommender Systems
• Claim Calibration

Innovation

• claim calibration
• reproducibility
• recommender systems
• Claim Evidence Profile