Skip to content

feat(standard-tests): validate tool call chunks during streaming - #34707

Merged
Mason Daugherty (mdrxy) merged 14 commits into
masterfrom
mdrxy/tool-calls-streaming
Jun 11, 2026
Merged

feat(standard-tests): validate tool call chunks during streaming#34707
Mason Daugherty (mdrxy) merged 14 commits into
masterfrom
mdrxy/tool-calls-streaming

Conversation

@mdrxy

@mdrxy Mason Daugherty (mdrxy) commented Jan 12, 2026

Copy link
Copy Markdown
Member

As a LangChain user streaming a tool-calling model, I expect each streamed chunk to expose structured tool_call_chunk content blocks so I can render or process tool calls live, instead of waiting for the final aggregated message.

This adds tool_call_streaming to ModelProfile and uses it in the standard chat-model tool-calling tests. When a model profile opts in, test_tool_calling and test_tool_calling_async now validate that at least one streamed chunk includes a tool_call_chunk block via content_blocks, while preserving the existing final-message validation.

This keeps the contract profile-gated so providers can opt in once their streaming chunk shape is verified. This PR opts in the providers verified by smoke testing with straightforward profile coverage: OpenAI, Anthropic, Fireworks, HuggingFace, OpenRouter, DeepSeek, and xAI. The generated profile artifacts are refreshed so runtime profiles expose the new capability flag.

Perplexity Responses also passed the smoke test, but its current profile data is for the sonar family while the Responses smoke path used a routed model string. That profile strategy is left as follow-up. MistralAI currently streams .tool_call_chunks, but its content-block translator exposes a complete tool_call block instead of tool_call_chunk, so it also stays out of this flag until that integration is fixed.

@github-actions github-actions Bot added standard-tests Related to the package `standard-tests` feature For PRs that implement a new feature; NOT A FEATURE REQUEST labels Jan 12, 2026
Comment thread libs/standard-tests/langchain_tests/integration_tests/chat_models.py Outdated
Comment thread libs/standard-tests/langchain_tests/unit_tests/chat_models.py Outdated
Comment thread libs/standard-tests/langchain_tests/integration_tests/chat_models.py Outdated
@github-actions github-actions Bot added integration PR made that is related to a provider partner package integration core `langchain-core` package issues & PRs model-profiles `langchain-model-profiles` package issues & PRs anthropic `langchain-anthropic` package issues & PRs openai `langchain-openai` package issues & PRs labels Jan 13, 2026
@codspeed-hq

codspeed-hq Bot commented Jan 13, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 22 untouched benchmarks
⏩ 14 skipped benchmarks1


Comparing mdrxy/tool-calls-streaming (b7db58f) with master (6199525)

Open in CodSpeed

Footnotes

  1. 14 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@mdrxy Mason Daugherty (mdrxy) changed the title feat(standard-tests): add streaming tool call tests for chat models Feb 21, 2026
@github-actions github-actions Bot added feature For PRs that implement a new feature; NOT A FEATURE REQUEST and removed feature For PRs that implement a new feature; NOT A FEATURE REQUEST labels Feb 21, 2026
@github-actions github-actions Bot added the size: S 50-199 LOC label Mar 9, 2026
@mdrxy

Copy link
Copy Markdown
Member Author

Local smoke-test summary for the provider follow-up:

Checked whether a streamed tool call produces at least one per-chunk content_blocks entry with type == "tool_call_chunk", and whether the final aggregate still contains a valid tool_calls entry. I ran both default output and output_version="v1" where supported.

Passed:

  • Fireworks (accounts/fireworks/models/kimi-k2p6)
  • HuggingFace via routed Fireworks provider (moonshotai/Kimi-K2.6)
  • OpenRouter (openai/gpt-4o-mini)
  • Perplexity Responses API (openai/gpt-5.5, use_responses_api=True)
  • DeepSeek (deepseek-chat)
  • xAI (grok-4-fast-reasoning)

Did not pass this PR's exact content_blocks criterion:

  • MistralAI (mistral-large-latest): streamed chunks populate .tool_call_chunks, but content_blocks exposes a complete tool_call block instead of tool_call_chunk. This looks fixable in the Mistral content-block translator, so I left Mistral out of tool_call_streaming for this PR.

This PR remains scoped to enabling the flag for OpenAI and Anthropic only; the broader provider list can be follow-up work after profile decisions/translator fixes.

@mdrxy

Copy link
Copy Markdown
Member Author

Scope update: based on the smoke-test results above, this PR now also opts in the smoke-passing providers instead of leaving them for follow-up.

Added tool_call_streaming profile augmentations and regenerated profile artifacts for:

  • Fireworks
  • HuggingFace
  • OpenRouter
  • DeepSeek
  • xAI
  • Perplexity, limited to the smoke-tested Responses model (openai/gpt-5.5) rather than the whole default sonar family

MistralAI remains excluded because it does not currently satisfy this PR's exact content_blocks contract.

@mdrxy

Copy link
Copy Markdown
Member Author

Follow-up to the scope update: Perplexity is now left out of this PR. Perplexity Responses passed the smoke test, but the generated Perplexity profile data currently covers the sonar family while the smoke path used a routed Responses model string. Keeping that profile strategy as follow-up avoids adding an awkward openai/gpt-5.5 entry to Perplexity profiles here.

@github-actions github-actions Bot added deepseek `langchain-deepseek` package issues & PRs fireworks `langchain-fireworks` package issues & PRs huggingface `langchain-huggingface` package issues & PRs openrouter `langchain-openrouter` package issues & PRs perplexity `langchain-perplexity` package issues & PRs size: L 500-999 LOC xai `langchain-xai` package issues & PRs and removed size: S 50-199 LOC labels Jun 11, 2026
@mdrxy
Mason Daugherty (mdrxy) merged commit 4388036 into master Jun 11, 2026
231 of 233 checks passed
@mdrxy
Mason Daugherty (mdrxy) deleted the mdrxy/tool-calls-streaming branch June 11, 2026 02:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

anthropic `langchain-anthropic` package issues & PRs core `langchain-core` package issues & PRs deepseek `langchain-deepseek` package issues & PRs feature For PRs that implement a new feature; NOT A FEATURE REQUEST fireworks `langchain-fireworks` package issues & PRs huggingface `langchain-huggingface` package issues & PRs integration PR made that is related to a provider partner package integration internal model-profiles `langchain-model-profiles` package issues & PRs openai `langchain-openai` package issues & PRs openrouter `langchain-openrouter` package issues & PRs perplexity `langchain-perplexity` package issues & PRs size: L 500-999 LOC standard-tests Related to the package `standard-tests` xai `langchain-xai` package issues & PRs

2 participants