perf(tools): prefetch streamed filesystem calls

[mpscholten]

Summary

• add a provider-neutral streamed tool interpreter to the loop, with the completed ToolCall remaining authoritative
• overlap filesystem work with the remaining argument stream for read_file, list_dir, search_replace, and apply_patch
• preserve ordinary approval, resource scheduling, rich success/failure classification, retry, and cancellation semantics
• fall back to normal execution on an incomplete or ambiguous path, late completion, stale data, changed arguments, denied access, parser failure, or cancellation
• share path-prefix, fingerprint, Git-index, bounded-concurrency, cancellation, and formatting helpers rather than duplicating them in each tool

Design

Every tool now runs through a small StreamedTool fold:

data ToolInput = ToolPrefix Text | ToolDone Text

data StreamedTool = forall s args. StreamedTool
    { streamedStart :: IO s
    , streamedInterpret :: s -> ToolInput -> IO (Either (args, s) s)
    , streamedConsume :: ToolCall -> (Text -> IO ()) -> args -> s -> IO ToolResult
    , streamedClose :: s -> IO ()
    }

The provider adapter emits provider-neutral argument events. ToolSpeculationRuntime correlates provider item IDs, output indexes, and call IDs, owns one scoped worker per call, and retains work only for calls in the final response. The final call is checked before a prepared result can be consumed.

Filesystem interpreters can start only when a path is complete or a streamed prefix uniquely identifies a Git-indexed workspace path. Prepared reads are bounded (four concurrent files, 16 MiB maximum), preserve the normal PDF/NUL checks, and are revalidated with a file fingerprint immediately before use. Approval and resource claims still run before consumption, including external-root access.

Correctness audit

This revision fixes and covers:

• positive subranges within a prefetched default 1,000-line window
• negative offsets and ranges outside the prefetched window falling back instead of returning a false hit
• avoiding a read beyond the selected 1,000th newline
• matching normal binary detection by checking the first 8 KiB, including a NUL later in a larger file
• changed/replaced files and final arguments invalidating prepared results
• a denied external path prompting exactly once
• retries, conflicting stream aliases, abandoned calls, reset/shutdown cancellation, and bounded worker capacity
• semantic handler failures remaining failures even when rendered text is customized
• repeated selected native-agent output staying within its configured cap

The edge cases are covered in dedicated speculation, interpreter, filesystem, loop, dialect, provider, and CLI tests.

Benchmarks

Built with GHC 9.10.3 and -O2; each result is forced and measured with +RTS -T -N1. Fixtures are created outside the measured interval, while the configured tail deliberately models argument-streaming time still available for overlap. All compared modes produced identical checksums.

nix develop -c cabal build --offline --enable-optimization=2 \
  agent-core:bench:speculative-read-file-bench \
  agent-cli:bench:speculative-tools-bench

Isolated read_file

15 samples with a 10 ms remaining stream tail:

Mode	File	Wall median	CPU median	Allocation	Hits / misses
baseline	1 MiB	11.386 ms	1.240 ms	4,134,000 B	0 / 0
completed speculation	1 MiB	10.383 ms	1.663 ms	4,361,104 B	15 / 0
late prefix / fallback	1 MiB	11.842 ms	1.979 ms	5,358,792 B	0 / 15
baseline	8 MiB	17.856 ms	7.682 ms	30,680,312 B	0 / 0
completed speculation	8 MiB	10.331 ms	7.744 ms	30,907,544 B	15 / 0
late prefix / fallback	8 MiB	17.868 ms	7.968 ms	31,906,320 B	0 / 15

Completed speculation reduced median wall time by 8.8% at 1 MiB and 42.1% at 8 MiB. The representative 8 MiB miss path remained effectively at baseline.

Mixed coding turn

The benchmark replays list_dir, four read_file calls, search_replace, and apply_patch against 8 MiB edit targets (31 samples):

Run	Tail	Baseline p50	Speculative p50	Change
primary	10 ms	64.939 ms	59.242 ms	8.8% faster
longer overlap	50 ms	109.452 ms	100.515 ms	8.2% faster
repeat	10 ms	71.714 ms	58.217 ms	18.8% faster

Speculation allocated about 2.9 MiB more per mixed turn (roughly 2.9%). Median CPU varied from +0.3 ms to +7.8 ms, as expected for work moved ahead and overlapped rather than eliminated.

Validation

• synchronized with master at cd37be69; it is an ancestor of head ea29944f
• exact 14-library multi-package GHCi load: 450 modules
• optimized tests: 2,948 examples, 0 failures, 1 expected pending live-auth example
  • agent-core: 542
  • agent-cli: 1,459
  • agent-tui: 195
  • Codex/Grok dialects: 15 / 49
  • Responses/Gemini/MCP/Claude SDK/OpenAI/agent-claude: 83 / 48 / 95 / 91 / 326 / 45
• packages/agent-core/package.nix regenerated and checked
• git diff --check