Subgroup errors
An overall score can improve while a smaller population receives worse predictions. You will examine errors within declared segments and report the amount of outcome evidence behind each estimate. A segment with one observed event has fragile recall, even when its displayed value looks precise. Missing outcomes make the comparison harder, so evaluate only mature labels and keep incomplete windows visible. The slices in this lesson describe fictional customer groups and model residuals. They support diagnosis, not a claim of legal fairness or a causal explanation. Use the business requirement to choose the comparison, then inspect uncertainty before proposing a change.
segment_recall = np.array([recall_score(monitor.loc[monitor.mature & (monitor.segment == s), "label"], monitor.loc[monitor.mature & (monitor.segment == s), "score"] >= 0.5, zero_division=0) for s in ["new", "repeat"]])
Separate error rates reveal differences that the overall score can conceal.