---
description: Polars DataFrame conventions
globs: ["**/*.py","notebooks/**/*.ipynb"]
alwaysApply: false
---

# Polars

- Check the installed Polars version (`pl.__version__`) and follow its API; deprecated names change between majors.
- Prefer the lazy API: start with `pl.scan_parquet`/`scan_csv`/`scan_ndjson`, build the query, call `.collect()` once at the end. Use `.collect(engine="streaming")` for larger-than-memory data if the version supports it.
- Use expressions, not Python loops: `df.with_columns((pl.col("price") * pl.col("qty")).alias("total"))`, `pl.when(...).then(...).otherwise(...)`, `group_by(...).agg(...)`.
- Avoid `map_elements`/`apply` (row-wise Python). If unavoidable, say why and set `return_dtype`.
- Declare schemas/dtypes when reading (`schema_overrides=`) and cast early; use `pl.Categorical`/`pl.Enum` for low-cardinality strings and proper `pl.Datetime(time_zone="UTC")`.
- Filter and select columns as early as possible (projection/predicate pushdown).
- Joins: specify `how` and `validate="m:1"` etc. to catch duplicate keys; use `join_asof` for time-series alignment.
- Don't convert to pandas unless a downstream library requires it (`.to_pandas()` at the very end).
- Inspect plans with `lf.explain()` when performance matters.
- Write outputs as Parquet with compression; partition large outputs by a column.

## Done when
Tests on small fixture frames pass (`polars.testing.assert_frame_equal`), and the pipeline runs end to end on sample data.