---
description: pandas data analysis conventions
globs: ["**/*.py","**/*.ipynb"]
alwaysApply: false
---

# pandas

- Assume Copy-on-Write semantics (default in pandas 3, opt-in via `pd.options.mode.copy_on_write = True` in 2.x). Never rely on chained assignment (`df[df.a > 0]["b"] = 1`); use `df.loc[mask, "b"] = 1`.
- Vectorize: column arithmetic, `.where`, `np.select`, `.str`/`.dt` accessors, `merge`, `groupby().agg()`. No `iterrows` or row-wise `apply` on large frames.
- Readable pipelines via method chaining with `.assign()`, `.query()`, `.pipe()`; one step per line.
- Set dtypes on read (`dtype=`, `parse_dates=`, `dtype_backend="pyarrow"` where supported). Use `category` for low-cardinality strings and nullable types (`Int64`, `string`) to avoid object columns.
- Datetimes are timezone-aware (`utc=True`), converted for display only.
- Joins: specify `how` and `validate="one_to_one"`/`"many_to_one"`; check row counts before/after.
- `groupby(..., observed=True)` for categoricals; named aggregations (`agg(total=("amount", "sum"))`).
- Handle missing data explicitly (`isna`, `fillna` with a reason), not by silently dropping.
- Keep `inplace=True` out of new code.
- For data that doesn't fit in memory, sample, chunk (`chunksize`), or switch to Polars/DuckDB, and say so.
- Save intermediate data as Parquet, not CSV or pickle.

## Done when
Assertions on shapes and key invariants (no duplicate keys, expected columns) pass, and tests use `pd.testing.assert_frame_equal` on small fixtures.