You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A Comet Native Executor cluster (#7289) only runs queries. A plan with a DataWritingCommandExec or a V2 write command is left to Spark, so INSERT, CTAS and DataFrame.write don't run on the cluster.
Describe the potential solution
Write the files on the executors and commit on the driver, like Comet's split Iceberg writer (IcebergWriteExec and IcebergCommitExec): the cluster returns the written files' metadata as the query's result, and the driver runs the commit protocol.
Additional context
#7288 lists writes as out of scope at first. CometNativeExecutors.applies skips write commands today.
Activity