Large files can fill your computer's memory. df.memory_usage(deep=True) shows the bytes per column; without deep=True, text columns are counted as pointers only and look far smaller than they are.
Ways to shrink a table:
- Text with few distinct values (regions, channels, bike types):
astype("category") stores each value once plus a small code per row.
- Whole numbers:
pd.to_numeric(col, downcast="integer") picks the smallest type that fits, such as int8 for values up to 127.
- Whole numbers with gaps:
int can't hold NaN, so pandas stores them as decimals. The nullable types "Int8", "Int16", "Int64" (capital I) can.
- Read less:
read_csv(usecols=[...], dtype={...}) loads only the columns you need, already in the right type.
A category with almost as many distinct values as rows saves nothing, so check nunique() first.