feat(database): add an infrastructure-error classifier and read-retry helper - #4863
feat(database): add an infrastructure-error classifier and read-retry helper#4863d-cs wants to merge 1 commit into
Conversation
|
WalkthroughThe database package adds Prisma and connectivity-error classification. It adds Merge Risk: 🟡 Moderate · up to The new opt-in retry helper can produce invalid retry delays from malformed configuration and leaves replay safety and retry-budget enforcement to callers, which could cause incorrect timing, duplicate writes, or amplified database load when adopted. Its disabled-by-default behavior limits immediate production exposure, but these safeguards require explicit owner attention before merge. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Full details: Description checkExplanation The description clearly explains the implementation, retry behavior, safety requirements, and objectives. It omits most required template sections, including the issue reference, checklist, testing steps, changelog, and screenshots.
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://cdn.statically.io/img/avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Team
Run ID: 514a4dd0-9520-4fd6-ae6f-7f688e06c148
📒 Files selected for processing (5)
internal-packages/database/src/index.tsinternal-packages/database/src/infraError.test.tsinternal-packages/database/src/infraError.tsinternal-packages/database/src/infraRetry.test.tsinternal-packages/database/src/infraRetry.ts
Included review availability: Your plan provides up to 10 included reviews per hour; 8 remain after this review.
📜 Review details
⏰ Context from checks skipped due to timeout. (32)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (11, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (24, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (4, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (6, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (8, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (18, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (12, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (21, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (17, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (5, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (15, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (23, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (22, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (7, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (20, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (10, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (9, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (19, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (1, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (13, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (3, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (16, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (14, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (2, 24)
- GitHub Check: e2e-webapp / 🧪 E2E Tests: Webapp (2, 2)
- GitHub Check: e2e-webapp / 🧪 E2E Tests: Webapp (1, 2)
- GitHub Check: typecheck / typecheck
- GitHub Check: fk-cascade-guard / fk-cascade-guard
- GitHub Check: runops-guard / runops-guard
- GitHub Check: internal / 🧪 Unit Tests: Internal
- GitHub Check: code-quality / code-quality
- GitHub Check: Analyze (javascript-typescript)
🧰 Additional context used
📓 Path-based instructions (9)
We use vitest exclusively. **Never mock anything** - use testcontainers instead.
📄 CodeRabbit inference engine (AGENTS.md)
Files:
internal-packages/database/src/infraError.test.tsinternal-packages/database/src/infraRetry.test.ts
**Prefer static imports over dynamic imports.** Only use dynamic `import()` when:
📄 CodeRabbit inference engine (AGENTS.md)
Files:
internal-packages/database/src/index.tsinternal-packages/database/src/infraError.test.tsinternal-packages/database/src/infraRetry.tsinternal-packages/database/src/infraRetry.test.tsinternal-packages/database/src/infraError.ts
Add crumbs as you write code — not just when debugging. Mark lines with
📄 CodeRabbit inference engine (AGENTS.md)
Files:
internal-packages/database/src/index.tsinternal-packages/database/src/infraError.test.tsinternal-packages/database/src/infraRetry.tsinternal-packages/database/src/infraRetry.test.tsinternal-packages/database/src/infraError.ts
Use `$replica` from `~/db.server` for read-heavy queries in the webapp.
📄 CodeRabbit inference engine (internal-packages/database/CLAUDE.md)
Files:
internal-packages/database/src/index.tsinternal-packages/database/src/infraError.test.tsinternal-packages/database/src/infraRetry.tsinternal-packages/database/src/infraRetry.test.tsinternal-packages/database/src/infraError.ts
Use Prisma for database interactions in internal-packages/database with PostgreSQL
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Files:
internal-packages/database/src/index.tsinternal-packages/database/src/infraError.test.tsinternal-packages/database/src/infraRetry.tsinternal-packages/database/src/infraRetry.test.tsinternal-packages/database/src/infraError.ts
Use vitest for all tests in the Trigger.dev repository
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Files:
internal-packages/database/src/infraError.test.tsinternal-packages/database/src/infraRetry.test.ts
Use function declarations instead of default exports
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Files:
internal-packages/database/src/index.tsinternal-packages/database/src/infraError.test.tsinternal-packages/database/src/infraRetry.tsinternal-packages/database/src/infraRetry.test.tsinternal-packages/database/src/infraError.ts
Use types over interfaces for TypeScript
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Files:
internal-packages/database/src/index.tsinternal-packages/database/src/infraError.test.tsinternal-packages/database/src/infraRetry.tsinternal-packages/database/src/infraRetry.test.tsinternal-packages/database/src/infraError.ts
When creating or editing OTEL metrics (counters, histograms, gauges), ensure metric attributes have low cardinality by using only enums, booleans, bounded error codes, or bounded shard IDs
📄 CodeRabbit inference engine (.cursor/rules/otel-metrics.mdc)
Files:
internal-packages/database/src/index.tsinternal-packages/database/src/infraError.test.tsinternal-packages/database/src/infraRetry.tsinternal-packages/database/src/infraRetry.test.tsinternal-packages/database/src/infraError.ts
🧠 Learnings (3)
📚 Learning: 2026-06-04T18:16:35.386Z
Learnt from: nicktrn
Repo: triggerdotdev/trigger.dev PR: 3836
File: apps/supervisor/src/backpressure/backpressureMonitor.ts:3-5
Timestamp: 2026-06-04T18:16:35.386Z
Learning: When reviewing TypeScript in this repo, apply the rule “prefer type aliases over interfaces” only to data/object shapes and union/intersection type modeling. If an interface is being used as a behavioral contract for collaborators to implement (e.g., method-shape interfaces that define required behavior, such as `BackpressureLogger` / `BackpressureSignalSource` in `apps/supervisor/src/backpressure/backpressureMonitor.ts`), keep it as an `interface` and do not flag it as a type-alias-vs-interface violation.
Applied to files:
internal-packages/database/src/infraRetry.ts
📚 Learning: 2026-05-18T08:21:27.694Z
Learnt from: d-cs
Repo: triggerdotdev/trigger.dev PR: 3632
File: apps/webapp/sentry.server.ts:4-21
Timestamp: 2026-05-18T08:21:27.694Z
Learning: When handling Prisma errors for P1001 ("Can't reach database server"), do not assume it only appears under a single property name. Prisma may surface P1001 via either `PrismaClientKnownRequestError` (`err.code === "P1001"`, e.g., mid-query connection drops) or `PrismaClientInitializationError` (`err.errorCode === "P1001"`, e.g., client startup connection failure). To reliably detect the condition, check `err.code === "P1001" || err.errorCode === "P1001"`, and avoid review rules that would incorrectly flag `err.code === "P1001"` as unreachable/never-matching.
Applied to files:
internal-packages/database/src/infraError.ts
📚 Learning: 2026-05-18T08:21:27.694Z
Learnt from: d-cs
Repo: triggerdotdev/trigger.dev PR: 3632
File: apps/webapp/sentry.server.ts:4-21
Timestamp: 2026-05-18T08:21:27.694Z
Learning: When handling Prisma error P1001 ("Can't reach database server") in TypeScript, don’t assume a single error shape. Prisma can surface P1001 via two different error classes/fields: `PrismaClientKnownRequestError` exposes it as `err.code === "P1001"` (common during mid-query connection drops), while `PrismaClientInitializationError` exposes it as `err.errorCode === "P1001"` (common on client startup failure). Therefore, predicates should use `err.code === "P1001" || err.errorCode === "P1001"`. Do not flag `err.code === "P1001"` as “unreachable/never matches,” as it is expected in production.
Applied to files:
internal-packages/database/src/infraError.ts
e35afcf to
f4f22da
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
internal-packages/database/src/infraRetry.ts (1)
40-43: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd the required crumb marker during development.
This new
withInfraRetrypath has no//@Crumbsmarker or `// `#region` `@crumbsblock. Add an approved existing marker, then strip it withagentcrumbs stripbefore merge. Do not invent a namespace.As per coding guidelines, “
**/*: Add crumbs as you write code — not just when debugging.”Source: Coding guidelines
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Team
Run ID: c74c9ed3-0c78-4d7e-aca6-c392d1c4096b
📒 Files selected for processing (1)
internal-packages/database/src/infraRetry.ts
Included review availability: Your plan provides up to 10 included reviews per hour; 8 remain after this review.
📜 Review details
⏰ Context from checks skipped due to timeout. (32)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (23, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (20, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (19, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (24, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (22, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (17, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (16, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (8, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (14, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (18, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (21, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (7, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (12, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (11, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (10, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (13, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (9, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (4, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (15, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (5, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (6, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (3, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (1, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (2, 24)
- GitHub Check: e2e-webapp / 🧪 E2E Tests: Webapp (2, 2)
- GitHub Check: fk-cascade-guard / fk-cascade-guard
- GitHub Check: typecheck / typecheck
- GitHub Check: e2e-webapp / 🧪 E2E Tests: Webapp (1, 2)
- GitHub Check: runops-guard / runops-guard
- GitHub Check: internal / 🧪 Unit Tests: Internal
- GitHub Check: code-quality / code-quality
- GitHub Check: Analyze (javascript-typescript)
🧰 Additional context used
📓 Path-based instructions (7)
**Prefer static imports over dynamic imports.**
📄 CodeRabbit inference engine (AGENTS.md)
Files:
internal-packages/database/src/infraRetry.ts
Add crumbs as you write code — not just when debugging.
📄 CodeRabbit inference engine (AGENTS.md)
Files:
internal-packages/database/src/infraRetry.ts
Use `$replica` from `~/db.server` for read-heavy queries in the webapp.
📄 CodeRabbit inference engine (internal-packages/database/CLAUDE.md)
Files:
internal-packages/database/src/infraRetry.ts
Use Prisma for database interactions in internal-packages/database with PostgreSQL
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Files:
internal-packages/database/src/infraRetry.ts
Use function declarations instead of default exports
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Files:
internal-packages/database/src/infraRetry.ts
Use types over interfaces for TypeScript Avoid using enums; prefer string unions or const objects instead
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Files:
internal-packages/database/src/infraRetry.ts
When creating or editing OTEL metrics (counters, histograms, gauges), ensure metric attributes have low cardinality by using only enums, booleans, bounded error codes, or bounded shard IDs Do not use high-cardinality attributes in OTEL metr...
📄 CodeRabbit inference engine (.cursor/rules/otel-metrics.mdc)
Files:
internal-packages/database/src/infraRetry.ts
🧠 Learnings (1)
📚 Learning: 2026-06-04T18:16:35.386Z
Learnt from: nicktrn
Repo: triggerdotdev/trigger.dev PR: 3836
File: apps/supervisor/src/backpressure/backpressureMonitor.ts:3-5
Timestamp: 2026-06-04T18:16:35.386Z
Learning: When reviewing TypeScript in this repo, apply the rule “prefer type aliases over interfaces” only to data/object shapes and union/intersection type modeling. If an interface is being used as a behavioral contract for collaborators to implement (e.g., method-shape interfaces that define required behavior, such as `BackpressureLogger` / `BackpressureSignalSource` in `apps/supervisor/src/backpressure/backpressureMonitor.ts`), keep it as an `interface` and do not flag it as a type-alias-vs-interface violation.
Applied to files:
internal-packages/database/src/infraRetry.ts
🔇 Additional comments (1)
internal-packages/database/src/infraRetry.ts (1)
1-7: LGTM!Also applies to: 16-39
Adds a shared classifier (isInfrastructureError / looksLikeConnectivityError) recognising connection-blip failures (P1001/P1002/P1008/P1017, ECONNRESET, "connection terminated", "server has closed the connection"), and withInfraRetry — a retry helper gated by an enabled kill-switch (default off) and the existing TokenBucketRetryBudget. Only for operations safe to run more than once (reads, or writes made idempotent); it never authorises retrying a bare non-idempotent write. TRI-13553
f4f22da to
0436e22
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Team
Run ID: 94718a1c-0ea8-4b7b-84bc-2dbd74768dda
📒 Files selected for processing (2)
internal-packages/database/src/infraRetry.test.tsinternal-packages/database/src/infraRetry.ts
Included review availability: Your plan provides up to 10 included reviews per hour; 8 remain after this review.
📜 Review details
⏰ Context from checks skipped due to timeout. (32)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (24, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (9, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (12, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (11, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (10, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (17, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (23, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (8, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (19, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (13, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (15, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (18, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (7, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (4, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (21, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (5, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (14, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (20, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (2, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (1, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (22, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (6, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (16, 24)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (3, 24)
- GitHub Check: runops-guard / runops-guard
- GitHub Check: typecheck / typecheck
- GitHub Check: internal / 🧪 Unit Tests: Internal
- GitHub Check: e2e-webapp / 🧪 E2E Tests: Webapp (1, 2)
- GitHub Check: e2e-webapp / 🧪 E2E Tests: Webapp (2, 2)
- GitHub Check: fk-cascade-guard / fk-cascade-guard
- GitHub Check: code-quality / code-quality
- GitHub Check: Analyze (javascript-typescript)
🧰 Additional context used
📓 Path-based instructions (9)
We use vitest exclusively.
📄 CodeRabbit inference engine (AGENTS.md)
Files:
internal-packages/database/src/infraRetry.test.ts
**Prefer static imports over dynamic imports.**
📄 CodeRabbit inference engine (AGENTS.md)
Files:
internal-packages/database/src/infraRetry.test.tsinternal-packages/database/src/infraRetry.ts
Add crumbs as you write code — not just when debugging.
📄 CodeRabbit inference engine (AGENTS.md)
Files:
internal-packages/database/src/infraRetry.test.tsinternal-packages/database/src/infraRetry.ts
Use `$replica` from `~/db.server` for read-heavy queries in the webapp.
📄 CodeRabbit inference engine (internal-packages/database/CLAUDE.md)
Files:
internal-packages/database/src/infraRetry.test.tsinternal-packages/database/src/infraRetry.ts
Use Prisma for database interactions in internal-packages/database with PostgreSQL
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Files:
internal-packages/database/src/infraRetry.test.tsinternal-packages/database/src/infraRetry.ts
Use vitest for all tests in the Trigger.dev repository
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Files:
internal-packages/database/src/infraRetry.test.ts
Use function declarations instead of default exports
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Files:
internal-packages/database/src/infraRetry.test.tsinternal-packages/database/src/infraRetry.ts
Use types over interfaces for TypeScript Avoid using enums; prefer string unions or const objects instead
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Files:
internal-packages/database/src/infraRetry.test.tsinternal-packages/database/src/infraRetry.ts
When creating or editing OTEL metrics (counters, histograms, gauges), ensure metric attributes have low cardinality by using only enums, booleans, bounded error codes, or bounded shard IDs Do not use high-cardinality attributes in OTEL metr...
📄 CodeRabbit inference engine (.cursor/rules/otel-metrics.mdc)
Files:
internal-packages/database/src/infraRetry.test.tsinternal-packages/database/src/infraRetry.ts
🧠 Learnings (1)
📚 Learning: 2026-06-04T18:16:35.386Z
Learnt from: nicktrn
Repo: triggerdotdev/trigger.dev PR: 3836
File: apps/supervisor/src/backpressure/backpressureMonitor.ts:3-5
Timestamp: 2026-06-04T18:16:35.386Z
Learning: When reviewing TypeScript in this repo, apply the rule “prefer type aliases over interfaces” only to data/object shapes and union/intersection type modeling. If an interface is being used as a behavioral contract for collaborators to implement (e.g., method-shape interfaces that define required behavior, such as `BackpressureLogger` / `BackpressureSignalSource` in `apps/supervisor/src/backpressure/backpressureMonitor.ts`), keep it as an `interface` and do not flag it as a type-alias-vs-interface violation.
Applied to files:
internal-packages/database/src/infraRetry.ts
| * ONLY wrap operations that are safe to run more than once: reads, or writes | ||
| * that have been made idempotent. Never wrap a bare non-idempotent write. | ||
| */ | ||
| export async function withInfraRetry<R>( |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Add the required development crumbs.
internal-packages/database/src/infraRetry.ts#L40-L40: add an approved//@Crumbsmarker or `#region `@crumbsblock for the retry flow.internal-packages/database/src/infraRetry.test.ts#L23-L23: add an approved crumb marker for the test flow.
As per coding guidelines: “Add crumbs as you write code” and strip them before merge with agentcrumbs strip.
📍 Affects 2 files
internal-packages/database/src/infraRetry.ts#L40-L40(this comment)internal-packages/database/src/infraRetry.test.ts#L23-L23
Source: Coding guidelines
| } | ||
| const low = Math.max(0, Math.min(backoffMinMs, backoffMaxMs)); | ||
| const high = Math.max(low, backoffMaxMs); | ||
| const delayMs = Math.round(low + random() * (high - low)); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Reject invalid delay inputs.
Line 75 accepts non-finite backoff bounds and out-of-range injected random values. backoffMaxMs: NaN produces delayMs: NaN. A random value of -1 or 2 with bounds 100 and 300 produces -100 or 500.
Validate finite backoff bounds. Reject or clamp random values outside [0, 1]. Add regression tests for these inputs.
1 participant
Summary
Adds two shared primitives to
@internal/databasefor surviving brief database connection blips: a classifier that recognises connectivity/infrastructure failures (as opposed to real query errors), andwithInfraRetry, a helper that retries an operation on those failures.Details
isInfrastructureError/looksLikeConnectivityErrorrecognise connection-level failures (unreachable server, closed connection, connection reset, connect timeouts) and deliberately do not match query or validation errors, so retries never mask a real bug.withInfraRetry(run, config)retriesrunon a classified infrastructure error with jittered backoff, gated by a shared token-bucket budget so a mass freeze cannot amplify into a retry storm. It has anenabledkill-switch (off by default) and reuses the existing retry-budget primitive.It must only wrap operations that are safe to run more than once (reads, or writes made idempotent); it deliberately provides no path that silently retries a non-idempotent write.