You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Comet Native Executors: support dynamic partition pruning #7370
A Comet Native Executor cluster (#7289) runs whole queries without a JVM. A scan with dynamic partition pruning filters its partitions by the result of a subquery (InSubqueryExec, SubqueryBroadcastExec) that today runs in Spark. The driver doesn't support those yet, so a query whose plan uses DPP can't run on the cluster.
Describe the potential solution
Run the pruning subquery before submitting the query, as the prototype does for scalar subqueries, and prune the scan's file partitions on the driver before it builds each partition's plan. A broadcast-reuse variant could instead run the subquery's build side as a stage of the same query.
Additional context
Scalar subqueries already run first on the driver and are passed to the cluster as literals; see withSubqueryResults in CometNativeExecutorsExec.scala. TPC-H doesn't use DPP, since its tables aren't partitioned.
Activity