Repository navigation
perf(models/pi-ai): parse streamed tool-call arguments incrementally - #2740
Conversation
🦋 Changeset detectedLatest commit: d75bbae The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://cdn.statically.io/img/avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| "agents": patch | ||
| --- | ||
|
|
||
| `agents/models/pi-ai` parses streamed chat-completions tool-call arguments incrementally, instead of re-parsing the whole argument text on every delta. |
There was a problem hiding this comment.
🟡 Changeset lacks required documentation link
The agents/models/pi-ai changeset omits the required documentation link, leaving the release note without a route to details.
| `agents/models/pi-ai` parses streamed chat-completions tool-call arguments incrementally, instead of re-parsing the whole argument text on every delta. | |
| `agents/models/pi-ai` parses streamed chat-completions tool-call arguments incrementally, instead of re-parsing the whole argument text on every delta. See [Models for pi-ai](https://github.com/cloudflare/agents/blob/main/docs/agents/models-pi-ai.md). |
Was this helpful? React with 👍 or 👎 to provide feedback.
| let parser = this.toolArguments.get(index); | ||
| if (parser === undefined) { | ||
| parser = new PartialArguments(); | ||
| this.toolArguments.set(index, parser); | ||
| } | ||
| block.arguments = parser.push(args); |
There was a problem hiding this comment.
🔍 Interleaved calls remain unsupported
When a new tool index appears, the assembler closes the previous call. Later fragments for that index are ignored, so interleaved calls remain incomplete.
Was this helpful? React with 👍 or 👎 to provide feedback.
|
✅ agents import sizes: no significant changes ( |
agents
@cloudflare/ai-chat
@cloudflare/codemode
hono-agents
@cloudflare/shell
@cloudflare/think
@cloudflare/voice
@cloudflare/worker-bundler
commit: |
The chat-completions wire in
agents/models/pi-ainow parses streamed tool-call arguments with a newPartialArgumentsclass instead of calling pi-ai'sparseStreamingJsonon the whole text after every delta. Partialargumentsstill update on every delta. The final arguments still come fromparseStreamingJsonincloseToolCall, so finished tool calls are unchanged.Fixes #2734. For every delta,
parseStreamingJsonruns aJSON.parsethat throws, rebuilds the string one character at a time inrepairJson, throws again, then runspartial-json. That's quadratic over a long argument, like a file passed towrite.PartialArgumentskeeps the lexer state (open containers, in-string, pending escape) between deltas. On each delta it closes the open string and containers and calls the nativeJSON.parseonce:tru), the closed text doesn't parse, and the previous snapshot stands until the next delta.parseStreamingJsonfor the rest of that call.partial-jsontrims it.writeargument in 6-char deltas (Node microbenchmark)The workload numbers come from replaying recorded Workers AI responses through
examples/next/harnesses/piinwrangler dev, with V8 allocation sampling turned on.pnpm vitest --project workers src/tests/models/ran. The newpartial-arguments.test.tsfeeds JSON documents in several chunk sizes. It checks that each run ends atJSON.parseof the whole text, and that every snapshot that changes matchesparseStreamingJsonof that prefix. It also covers escapes split across deltas and the repair fallback, plus a wire-level test that streams arguments one character at a time.