Before analysis, check that the data can be trusted. Each check is a question with a count as its answer:
- Duplicates:
df.duplicated().sum() counts rows that repeat an earlier row exactly; df["id"].duplicated().sum() counts repeated keys.
- References: every
member_id in the orders should exist in the members table. ~orders["member_id"].isin(members["member_id"]) finds the ones that don't (often called orphans).
- Ranges and order: amounts above zero, dates inside the period, no order before the member signed up.
- Required fields:
isna().sum() per column.
Collect the counts in one table so a reader sees every problem at once, and decide for each one: fix, drop, or keep and note. In code that runs on every new file, assert orders["order_id"].is_unique, "repeated order ids" stops the script before a bad file reaches a report.