Add pluggable tool-search for large MCP toolsets #766

Activity

1. JassonG-xt commented on Aug 11, 2026

   

   JassonG-xt

   on Aug 11, 2026

   More actions

   I spent some time tracing how tools actually reach the wire before commenting, so here's what I
   found and a proposed shape. Line references are against 64eec1b6 (v0.10.4).

   There is no shared seam to hook into

   The issue mentions "each provider's _prepare_tools()". That method turns out to be
   Anthropic-only; each provider assembles its tool payload through its own code path:

   Provider	Entry point
   Anthropic	_prepare_tools(), llm/provider/anthropic/llm_anthropic.py:1123
   OpenAI Responses	_tools_payload(), llm/provider/openai/responses.py:844
   Google	_google_available_tools(), llm/provider/google/llm_google_native.py:803

   There is also no cross-provider capability protocol to hang this off. supports_web_tools()
   exists only on the Anthropic classes (llm_anthropic.py:771, overridden in
   llm_anthropic_vertex.py:80); FastAgentLLM (llm/fastagent_llm.py:110), its protocol, and
   interfaces.py declare no capability methods at all. So "a single abstraction that bridges to
   each provider's mechanism" means introducing the seam rather than plugging into one.

   Current state, for the record. OpenAI's server-side tool_search (responses.py:93) is already
   wired, but it only triggers for provider-managed MCP attachments carrying defer_loading
   (responses.py:866), not for tools coming through MCPAggregator. On Anthropic there is
   nothing yet:

   $ grep -ri 'tool_search\|defer_loading' src/fast_agent/llm/provider/anthropic/
   $
   

   The display vocabulary, on the other hand, already exists: the remote_tool_search family,
   the "Deferred tool search" label, and the status text in tool_activity_presentation.py:33,
   plus streaming and replay handling in the Responses pipeline. The socket is there; the plug
   isn't.

   The constraint I think should drive the design

   llm_anthropic.py:1263 applies cache_control to the system prompt, and the log line notes it
   "caches tools+system in one block".

   That makes the obvious approach, re-selecting a task-relevant tool subset each turn, actively
   harmful. Any change to the tool list invalidates the tools+system prefix, and rebuilding that
   cache can cost more than the tool schemas you saved. It may also be why both vendors landed on
   "send everything, mark it deferred" instead of "send a subset".

   So whatever we build has to keep a stable resident set across turns within a session and
   push only the tail behind a lookup. Deferral selection needs to be deterministic per session,
   or it quietly loses money.

   Proposed shape

   Introduce the capability seam on the LLM base class, following the supports_web_tools()
   precedent, and dispatch to one of two backends before each provider serialises its payload:

   FastAgentLLM
     supports_deferred_tools() -> bool        # default False
     tool_deferral_backend()   -> Backend | None
   
   policy.split(tools, ctx) -> (resident, deferred)   # pure, deterministic per session
     |- NativeBackend : mark deferred tools, let the provider's own tool search serve them
     |                  (Anthropic tool search; OpenAI reuses the existing responses.py path)
     |- LocalBackend  : drop deferred tools from the payload, inject one `search_tools` meta-tool
                        backed by a dependency-free lexical index over name/description/params
   

   McpAgent.list_tools() keeps its current signature and semantics. The split happens below it, at
   the provider boundary, because that is the only layer that knows both what the model supports and
   where the cache breakpoint falls.

   LocalBackend is what keeps this useful when the provider in front of you has no native
   mechanism to bridge to. I'd keep it lexical rather than embedding-based: tool descriptions
   already share vocabulary with user requests, and pulling in a model download to route tool calls
   seems like a bad trade. If semantic matching later proves necessary, the privacy module
   establishes the pattern for an optional extra (find_spec probing in privacy/dependencies.py,
   pinned repo_id/revision in privacy/model_resolver.py).

   Relationship to #589

   @iqdoctor's discover / learn / execute proposal covers much of this ground. I'd rather treat
   LocalBackend as one implementation of that Phase 2 than as a competing design; the meta-tool
   surface is the same idea, scoped to the aggregator's own tools instead of a standalone proxy.
   Happy to align naming with #589 if that's preferred.

   Suggested sequencing

   1. Capability seam, policy, and measurement only, with no behaviour change. Report
      tool-schema token cost per request; commands/status_summaries.py:312 already shows the
      tiktoken pattern to follow. Without a baseline number none of the later phases can be argued
      for.
   2. LocalBackend, which benefits every provider.
   3. NativeBackend, Anthropic first, since OpenAI mostly reuses what's already there.

   Questions before I write any code

   1. Is adding capability methods to the LLM base class the shape you want, or would you rather
      keep this provider-local and accept the duplication?
   2. Should the resident/deferred split be configurable per agent (in the spirit of tools: in
      fastagent.config.yaml), or inferred purely from tool count and usage?
   3. Is a dependency-free lexical index acceptable in core, or should even that sit behind an
      extra?

   Happy to start with step 1 alone if that's the useful piece.