Skip to content

Comet Native Executors: report a query's metrics in the Spark UI #7373

Open

Description

@andygrove

What is the problem the feature request solves?

A query that runs on a Comet Native Executor cluster (#7289) shows in the Spark UI as a single CometNativeExecutorsExec with no metrics: no rows, bytes, spill or time per operator, and no view of the stages it ran as.

Describe the potential solution

Return Ballista's per-stage and per-operator metrics with the query's result, and map each stage's metrics back onto the Spark operators of the native block it ran, so the plan's metrics fill in as they would for Comet on Spark.

Additional context

#7288 lists Spark UI parity as out of scope at first.

Part of #7288.

Willingness to contribute

Yes. I can contribute this feature independently.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions