Repository navigation
Require pandera>=0.23 and un-xfail the dask series test - #1770
Open
Thanatos9404 wants to merge 1 commit into
Open
Thanatos9404 wants to merge 1 commit into
Thanatos9404 wants to merge 1 commit into
Conversation
Validating a dask series against a pandera SeriesSchema raised KeyError: 'dask' with pandera<0.23 and dask>=2025.1. Since dask-expr moved into dask, dask series live in `dask.dataframe.dask_expr`, and pandera before 0.23 only maps the old top-level `dask_expr` module name to its dask backend. SeriesSchema.validate looks the type up first, so the series path failed; DataFrameSchema maps over the partitions before any lookup, which is why dask dataframes were unaffected. The error also happens without Hamilton. pandera fixed the lookup in 0.23.0 (unionai-oss/pandera#1898). Set pandera>=0.23 for the pandera extra, the test and docs dependency groups, and the pandera integration test requirements. Remove the xfail from test_pandera_decorator_dask_series and add a test where only the last partition of a dask series breaks the schema. Fixes apache#1279
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1279
The
KeyError: 'dask'comes from pandera rather than import order in Hamilton. Since dask 2025.1, dask-expr ships inside dask, so a dask series is adask.dataframe.dask_expr._collection.Series. pandera before 0.23 maps only the old top-leveldask_exprmodule name to its dask backend, andSeriesSchema.validatelooks up the backend by type first, so it fails.DataFrameSchema.validatemaps over the partitions before any lookup, which is why only the series test broke. Calling pandera directly on a dask series fails the same way with pandera 0.22.1 and dask 2025.1.0. pandera fixed the lookup in 0.23.0 (thedaskbackend mapping added in unionai-oss/pandera#1898), which is why the xfailed test already passes in CI.Changes
pandera>=0.23for thepanderaextra and thedocsgroup, andpandera[dask]>=0.23in thetestgroup andtests/integrations/pandera/requirements.txt.test_pandera_decorator_dask_series.No code changes in Hamilton itself.
How I tested this
tests/integrations/panderaon Python 3.12:KeyError: 'dask', which is why the floor is needed.tests/test_default_data_quality.py,tests/function_modifiers/test_validation.pyandtests/test_end_to_end.py.uv pip compile pyproject.toml --extra pandera --group teststill resolves on Python 3.10 and 3.14, andprek run --filespasses on the changed files.Notes
pandera 0.23 itself requires pandas>=2.1.1, so the
panderaextra now implies that floor too. Plain Hamilton is not affected.If you need to keep supporting pandera < 0.23, I have a small alternative that validates dask series partition by partition in
PanderaSeriesSchemaValidatorinstead of raising the floor, and can open that instead.Checklist