Skip to content

Comet Native Executors: support dynamic partition pruning #7370

Open

Description

@andygrove

What is the problem the feature request solves?

A Comet Native Executor cluster (#7289) runs whole queries without a JVM. A scan with dynamic partition pruning filters its partitions by the result of a subquery (InSubqueryExec, SubqueryBroadcastExec) that today runs in Spark. The driver doesn't support those yet, so a query whose plan uses DPP can't run on the cluster.

Describe the potential solution

Run the pruning subquery before submitting the query, as the prototype does for scalar subqueries, and prune the scan's file partitions on the driver before it builds each partition's plan. A broadcast-reuse variant could instead run the subquery's build side as a stage of the same query.

Additional context

Scalar subqueries already run first on the driver and are passed to the cluster as literals; see withSubqueryResults in CometNativeExecutorsExec.scala. TPC-H doesn't use DPP, since its tables aren't partitioned.

Part of #7288.

Willingness to contribute

Yes. I can contribute this feature independently.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions