Spot outliers
An outlier is a value far from the rest. It may be an error (a price typed without its decimal point) or real (a bike kept out all afternoon). Find it first, then decide.
- IQR rule: with
q1 and q3 the 25th and 75th percentiles and iqr = q3 - q1, values above q3 + 1.5 * iqr or below q1 - 1.5 * iqr are flagged. Percentiles barely move when a few extreme values are added, so this works on skewed data.
- z-score:
(x - x.mean()) / x.std(); values beyond 3 are flagged. The extreme values themselves pull the mean and widen the standard deviation, so in skewed data the z-score can miss them.
- clip:
x.clip(lower, upper) caps values instead of removing rows, so totals and counts still add up.
- Within groups: what is extreme for a lamp is normal for a sofa. Compare each value with its own group, using
groupby(...).transform(...).
rentals["duration_min"].quantile([0.25, 0.5, 0.75, 0.99])
Most rides are short; the top percent is much longer.