A probability forecast of 0.8 makes a frequency claim, while a ranking only states which observation should come first. Calibration evaluates that frequency claim. You will fit a sigmoid calibrator using held-out folds, calculate Brier loss and inspect reliability points. Empty bins can be absent from the output, so don't assume every requested bin has an observation. A strictly increasing mapping preserves score order even when it changes probability errors. That distinction explains why discrimination can stay constant while calibration changes. Calibration itself needs data excluded from fitting the underlying estimator, and a separate test remains necessary for the final assessment.