Skip to content

fix(server): messages cut by a restart no longer stay marked as streaming - #28646

Open
Abhishek-B-R wants to merge 1 commit into
pingdotgg:mainfrom
Abhishek-B-R:fix/recovery-cancelled-items-stop-streaming
Open

Abhishek-B-R wants to merge 1 commit into
pingdotgg:mainfrom
Abhishek-B-R:fix/recovery-cancelled-items-stop-streaming

Conversation

@Abhishek-B-R

Copy link
Copy Markdown

Problem

When a server restart or shutdown cuts a run, recovery cancels its open turn items but leaves streaming: true on assistant_message, reasoning and proposed_plan items. The item ends up cancelled and still streaming, and nothing clears it later because the run is over. Clients build the assistant row from that flag, so a partial reply keeps rendering as live output.

Expected: a cancelled item is not streaming, as already happens on the other terminal paths.

Change

cancelledItem in ProviderRuntimeRecoveryService.ts now sets streaming: false on any item that has the field, using the same "streaming" in item check as RunExecutionService. Those three types are the only turn items with the field, and other items keep their shape.

Scope and approval

No triaged issue for this code path, under the very small, focused fix for an obvious bug route: a cancelled item cannot still be streaming, and the fix is one spread in one helper.

Related history, stated plainly so nobody has to dig for it:

Verification

Extended the existing "cancels the complete in-flight subtree" case in ProviderRuntimeRecoveryService.test.ts with one streaming item of each type.

cd apps/server && vp test run src/orchestration-v2/ProviderRuntimeRecoveryService.test.ts -t "cancels the complete in-flight subtree"

  • Before the fix: fails, each of the three items is cancelled with streaming: true.
  • After the fix: passes, all three are false.

To check the client side, I ran a throwaway test (not committed) that applied a running, streaming assistant_message to the real SQLite projection store, ran recovery.reconcile("startup"), read the row back from getThreadProjection, and passed it to the web deriveTimelineEntriesFromVisibleTurnItems:

stored item timeline message.streaming
Before cancelled, streaming: true true
After cancelled, streaming: false false

The recovery, regression, projection recovery and restart continuation test files pass (75 tests), and so does vp run check:changed.

Not checked: a real provider turn killed mid-stream in a running app, and the rendered result. From reading TimelineAssistantRows.tsx, a streaming row hides its timestamp and keeps ChatMarkdown in streaming mode, but I did not see that on screen. Mobile maps the flag the same way in threadActivity.ts; I read that and did not run it.

Claude Opus 5.5 via Claude Code

@github-actions github-actions Bot added vouch:unvouched PR author is not yet trusted in the VOUCHED list. size:XS 0-9 changed lines (additions + deletions). labels Nov 1, 2026
@macroscopeapp

macroscopeapp Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved at 812cc1a

Macroscope's review found this PR approvable — Recovery now clears the streaming flag on interrupted turn items already being cancelled, preventing partial replies from continuing to appear live. Existing cancellation rules and message contents are unchanged, keeping the impact narrowly bounded.

You can add or adjust custom eligibility rules. Learn more.

@coderabbitai

coderabbitai Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 8760d06 and 812cc1a.

Files selected for processing: 2
  • apps/server/src/orchestration-v2/ProviderRuntimeRecoveryService.test.ts
  • apps/server/src/orchestration-v2/ProviderRuntimeRecoveryService.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 3 remain after this review.

Walkthrough

Recovery cancellation now clears the streaming flag on cancelled turn items that have that property. The recovery test covers streaming assistant_message, reasoning, and proposed_plan items.

Changes: Recovery cancellation

Layer / File(s) Summary
Clear streaming state on cancelled items
apps/server/src/orchestration-v2/ProviderRuntimeRecoveryService.ts, apps/server/src/orchestration-v2/ProviderRuntimeRecoveryService.test.ts
Cancellation clears streaming when the item has that property. The test verifies this for assistant_message, reasoning, and proposed_plan items.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: juliusmarminge

Merge Risk: Minimal · up to 812cc

The recovery change correctly clears streaming state for cancelled items and is ready to merge.

Pre-merge checks: Passed 4
Check name Status Explanation
Title check Passed The title clearly and concisely describes the main fix: clearing the streaming state for messages interrupted by a server restart.
Description check Passed The description includes all required sections. It explains the problem, change, scope rationale, verification results, limitations, and agent details.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Finishing Touches
Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
@maria-rcks

maria-rcks commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator

Note

Written by gpt-6.1-sol on behalf of Maria

Verified in a real dev environment

verified as-is at 812cc1ae9075689a14da6f5bd993172b2fa7aee1 on a disposable linux vm. no code changes.

used repo Playwright 1.60.0 with /usr/bin/google-chrome, the running dev web client, and real Codex gpt-6.1-sol turns through the local proxy. added a local git project through the app and selected the model in the composer. ran one dev server at a time with isolated state.

  • reproduced on github/main (82b10d546c): interrupting a streaming reply with server shutdown left the assistant item cancelled with streaming: true after restart. its timestamp and copy control stayed absent.
  • on the pr, shutdown recovery cleared streaming, preserved the partial reply, and the client returned to idle.
  • separately saved a read-only sqlite snapshot during a real in-flight main turn and started the pr against a copy of that state. startup recovery cancelled the same 825-character reply with streaming: false. the matching captures below show its timestamp and copy control restored. no turn items were fabricated or edited.
  • sent a follow-up in the recovered thread: the real provider replied recovery follow-up works, completed normally, and persisted streaming: false.
  • started another long reply and clicked stop: the client showed Run interrupted · Stopped by user, preserved partial text, and persisted streaming: false.
  • ran the recovery, restart continuation, projection recovery, and regression suites through blacksmith: all 75 tests passed. the existing head CI checks also passed.

read-only runtime transcript (type | status | streaming | text length):

main after shutdown/restart:
assistant_message|cancelled|1|825
pr startup recovery of the same in-flight reply:
assistant_message|cancelled|0|825
pr independent shutdown:
assistant_message|cancelled|0|3622
pr follow-up:
assistant_message|completed|0
pr user stop:
assistant_message|completed|0
run_interrupt_result|interrupted

before, main with the partial reply still marked streaming:

main recovery leaves cancelled reply without timestamp or copy control

after, pr startup recovery of the identical partial reply:

pr recovery restores timestamp and copy control on the same cancelled reply

real streaming reply, server shutdown, reconnect, and idle state:

real codex reply interrupted by server shutdown and recovered to idle

neighboring stop interaction:

stop generation interrupts a real codex reply and preserves partial output

Not verified

native ios/android and electron shells, macos/windows, remote relay/tunnel connections, and other providers were not exercised on this linux vm. real interruption of streaming reasoning and proposed-plan items was not exercised; those types are covered by the passing focused recovery test. startup was checked with a copy of actual in-flight state, rather than a machine crash. automatic restart continuation was disabled for the matching comparison; its existing tests passed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XS 0-9 changed lines (additions + deletions). vouch:unvouched PR author is not yet trusted in the VOUCHED list.

2 participants