Skip to content

Require pandera>=0.23 and un-xfail the dask series test - #1770

Open
Thanatos9404 wants to merge 1 commit into
apache:mainfrom
Thanatos9404:fix/1279-require-pandera-0-23
Open

Thanatos9404 wants to merge 1 commit into
apache:mainfrom
Thanatos9404:fix/1279-require-pandera-0-23

Conversation

@Thanatos9404

Copy link
Copy Markdown

Fixes #1279

The KeyError: 'dask' comes from pandera rather than import order in Hamilton. Since dask 2025.1, dask-expr ships inside dask, so a dask series is a dask.dataframe.dask_expr._collection.Series. pandera before 0.23 maps only the old top-level dask_expr module name to its dask backend, and SeriesSchema.validate looks up the backend by type first, so it fails. DataFrameSchema.validate maps over the partitions before any lookup, which is why only the series test broke. Calling pandera directly on a dask series fails the same way with pandera 0.22.1 and dask 2025.1.0. pandera fixed the lookup in 0.23.0 (the dask backend mapping added in unionai-oss/pandera#1898), which is why the xfailed test already passes in CI.

Changes

  • pandera>=0.23 for the pandera extra and the docs group, and pandera[dask]>=0.23 in the test group and tests/integrations/pandera/requirements.txt.
  • Removed the xfail from test_pandera_decorator_dask_series.
  • Added a test where only the last of two partitions of a dask series breaks the schema, so the series validator is checked to compute every partition.

No code changes in Hamilton itself.

How I tested this

tests/integrations/pandera on Python 3.12:

  • pandera 0.22.1 with dask 2025.1.0: both dask series tests fail with KeyError: 'dask', which is why the floor is needed.
  • pandera 0.23.0 (the new minimum) with dask 2025.1.0: all pass.
  • pandera 0.34.1 with dask 2026.8.0: all pass, as do tests/test_default_data_quality.py, tests/function_modifiers/test_validation.py and tests/test_end_to_end.py.

uv pip compile pyproject.toml --extra pandera --group test still resolves on Python 3.10 and 3.14, and prek run --files passes on the changed files.

Notes

pandera 0.23 itself requires pandas>=2.1.1, so the pandera extra now implies that floor too. Plain Hamilton is not affected.

If you need to keep supporting pandera < 0.23, I have a small alternative that validates dask series partition by partition in PanderaSeriesSchemaValidator instead of raising the floor, and can open that instead.

Checklist

  • PR has an informative and human-readable title (this will be pulled into the release notes)
  • Changes are limited to a single goal (no scope creep)
  • Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future TODOs are captured in comments
  • Project documentation has been updated if adding/changing functionality. (No user-facing change beyond the dependency floor.)
Validating a dask series against a pandera SeriesSchema raised
KeyError: 'dask' with pandera<0.23 and dask>=2025.1. Since dask-expr
moved into dask, dask series live in `dask.dataframe.dask_expr`, and
pandera before 0.23 only maps the old top-level `dask_expr` module name
to its dask backend. SeriesSchema.validate looks the type up first, so
the series path failed; DataFrameSchema maps over the partitions before
any lookup, which is why dask dataframes were unaffected. The error
also happens without Hamilton. pandera fixed the lookup in 0.23.0
(unionai-oss/pandera#1898).

Set pandera>=0.23 for the pandera extra, the test and docs dependency
groups, and the pandera integration test requirements. Remove the xfail
from test_pandera_decorator_dask_series and add a test where only the
last partition of a dask series breaks the schema.

Fixes apache#1279

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant