Repository navigation
docs(skills): get Copilot's testing → test-servers hand-off to 100% (#2459) - #2477
Merged
Merged
Conversation
Copilot's testing -> test-servers hand-off on the pagination prompt missed because the first move was never `testing` (a grep, or an issue-triage detour) — never a loaded `testing` that failed to follow its pointer. So the description, not the body, was the lever: it now opens with writing a test or end-to-end/integration coverage of an MCP operation. Also tells a caller arriving at test-servers that every showcase-config field (maxPageSize included) is a createTestServerHttp option, and brings AGENTS.md's listing-cost figures up to 3,900/4,000. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: cliffhall <cliff@futurescale.com>
5 tasks
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The new guidance incorrectly implies raw showcase-config fields are valid constructor options.
Get a fresh assessment by requesting another Copilot review.
Review effort: Balanced
Findings: 1
Open (1)
What changed in this PR
Improves Copilot’s testing-to-test-server skill hand-off for MCP integration coverage.
Changes:
- Expands the testing skill’s trigger description.
- Adds in-process pagination/config guidance.
- Updates skill-listing budget documentation.
| File | Description |
|---|---|
AGENTS.md |
Updates listing-budget figures. |
.claude/skills/testing/SKILL.md |
Strengthens test-related triggers. |
.claude/skills/test-servers/SKILL.md |
Documents in-process configuration mechanics. |
…raw JSON fields (#2477 review) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: cliffhall <cliff@futurescale.com>
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2459
What was actually missing
The issue listed three levers. Before changing anything, I recorded Copilot's tool sequence on the pagination prompt (11 probes, same flags as
skills:eval). Every miss had the same shape: the first move was nevertesting. It was a plaingrep, or a detour throughissue-triage. None of the misses loadedtestingand then failed to follow its pointer. #2399 had already moved the pointer to the top of the body and told the reader to loadtest-serversbefore searching. So the body wasn't the bottleneck. The description was.Changes
testingdescription (lever 2): now opens with "Write, run, place and fix tests… Use when adding a test, or end-to-end or integration coverage of an MCP operation (listing, paginating or calling tools)". Probe after the change: 8/8 runs loadedtestingthentest-serversas their first two calls, before any grep.test-serversdescription) would spend listing budget on a first-move load that the chain case doesn't score.test-serversbody: a fifth in-process mechanic. This comes from the The testing → test-servers hand-off lands on the wrong half of the procedure #2264 read-through. A caller arriving from the pagination prompt was routed correctly to in-process HTTP but never told how to make that server paginate, and every probe had to grep for it. The body now says every showcase-config field is also acreateTestServerHttpoption:maxPageSizewithcreateNumberedTools, with theinspectorClient.test.tscase that does it. It also points toempty-cursor.test.tsfor spreading a whole showcase config into an in-process server. This is a body edit, so it has no effect on triggering.AGENTS.md: the listing-cost figures now read 3,900/4,000 (they said 3,729, which was already stale at 3,775). The "budget is not tight" sentence now says the next skill added will need a description trimmed or the budget raised.Measurements (Copilot CLI 1.0.88, Claude Code 2.1.281)
Full suite, RUNS=3:
pr-flow's "Open the PR for docs: re-align the H2 2026 roadmap with the published MCP roadmap (2026-08-22) and official extensions #2400, then keep getting Copilot to review it…" at 67%, and 40% at RUNS=5 on this branch against 100% onv2/main. I tracked it down: in the misses Claude's first move wasRead .git. The eval ran on a branch namedv2/chore/2459-…whose head commit says(#2459), andclaude -pinjects git status, so a prompt about docs: re-align the H2 2026 roadmap with the published MCP roadmap (2026-08-22) and official extensions #2400 sent the model to check which branch it was on. The same commit on a detached HEAD scored 100% (5/5), and 6/6 on a probe. So this is an artifact of running the eval on a feature branch for a different issue, not an effect of this diff.npm run verify:skillspasses at 3,900/4,000.npm run local:gateis green.🤖 Generated with Claude Code