Skip to content

Comet Native Executors: reuse Flight connections in the driver's client #7376

Open

Description

@andygrove

What is the problem the feature request solves?

The driver's Flight client (#7289) opens a new gRPC channel for every result partition it reads, even when many partitions live on the same few executors.

Describe the potential solution

Keep one channel per executor location in a process-wide cache in native/core/src/execution/native_executors/client.rs, and clone it for each partition. A tonic Channel is cheap to clone and multiplexes requests.

Additional context

This is a small efficiency gain, not a functional gap.

Part of #7288.

Willingness to contribute

Yes. I can contribute this feature independently.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions