Merge pull request #7170 from cloudflare/jasnell/streams-test-consolidation-4

Add digest streams suite; migrate existing DigestStream tests

Author: jasnell

src/tests/streams/AGENTS.md

@@ -1,8 +1,8 @@
 # src/tests/streams/
 
 Streams test suite, organized WPT-style: one subdirectory per functional area
-(`identity/`, `encoding/`, `compression/`, eventually `readable/`,
-`writable/`, `piping/`, ...). Every
+(`identity/`, `encoding/`, `compression/`, `digest/`, eventually
+`readable/`, `writable/`, `piping/`, ...). Every
 test here runs against **both** streams implementations — the legacy C++ one
 (`src/workerd/api/streams/`) and the TypeScript one
 (`src/per_isolate/webstreams/`) — to prove parity. A test that only makes

src/tests/streams/digest/AGENTS.md

@@ -0,0 +1,128 @@
+# crypto.DigestStream
+
+An informal specification of the Cloudflare-specific DigestStream (a
+WritableStream subclass computing a hash digest) as implemented in workerd,
+derived from — and kept in lockstep with — the test suite in this
+directory. **The tests are the normative artifact.** Both the C++
+implementation (`src/workerd/api/crypto/crypto.{h,c++}`) and the TypeScript
+implementation (`src/per_isolate/crypto/digest-stream.ts`, behind
+`typescript_implemented_streams`) are covered. Both drive the SAME native
+digest context (`utils.createDigestContext` → CRC/OpenSSL contexts and the
+WTF-8/toWellFormed string encoder), so hashing, string encoding, and byte
+counting are parity by construction; divergences live in the stream
+wrapper and promise plumbing. DigestStream is non-standard: no WPT exists.
+
+## Interface
+
+```webidl
+[Exposed=Worker]
+interface DigestStream : WritableStream {
+  constructor((DOMString or HashAlgorithm) algorithm,
+              optional DigestStreamOptions options = {});
+  readonly attribute Promise<ArrayBuffer> digest;  // memoized identity
+  readonly attribute bigint bytesWritten;
+};
+dictionary DigestStreamOptions { boolean toWellFormed = false; };
+```
+
+- Algorithms (case-insensitive over the whole name, string or `{name}`
+  object): `md5`, `SHA-1`, `SHA-256`, `SHA-384`, `SHA-512`, `crc32`,
+  `crc32c`, `crc64nvme`. Unknown names throw `NotSupportedError`
+  synchronously; the options bag is type-checked BEFORE the algorithm
+  lookup (primitive bags → TypeError "constructor parameter 2 is not of
+  type 'Options'."); `toWellFormed` is ToBoolean-coerced.
+- **Inheritance divergence (ledger #1):** instances are
+  `instanceof WritableStream` and the instance prototype chain is wired in
+  both, but only TypeScript's genuine `class extends` links the
+  CONSTRUCTOR's prototype to the WritableStream function; the C++ jsg
+  static chain does not. Subclassing via `class X extends
+  crypto.DigestStream` works in both.
+- `digest` and `bytesWritten` are brand-checked prototype accessors
+  (`digest` moves to a per-instance own property in the unflagged legacy
+  era); `Symbol.dispose` is brand-checked; `[object DigestStream]`
+  branding under `set_tostring_tag`.
+
+## Core semantics
+
+- **Write settlement:** the hash update runs inside write() (chunks are
+  consumed before the write settles — post-write mutation or detach cannot
+  change the digest; metadata comes from internal slots). Chunks:
+  ArrayBuffer, any view (offsets honored), strings; bare
+  SharedArrayBuffers reject (views onto them are accepted); everything
+  else rejects TypeError "DigestStream is a byte stream but received an
+  object of non-ArrayBuffer/ArrayBufferView/string type on its writable
+  side." Zero-length writes are no-ops.
+- **String encoding:** WTF-8 per chunk by default (lone surrogate = ED A0
+  80 — deliberately NOT TextEncoder's substitution; pinned for existing
+  digests). `toWellFormed: true` substitutes U+FFFD with
+  TextEncoderStream-style stateful pairing across chunks and a close-time
+  flush of a dangling lead. `bytesWritten` counts UTF-8 bytes (bigint in
+  every era).
+- **Digest promise:** memoized identity (same object every access, before
+  and after settling); resolves on close() with the digest ArrayBuffer;
+  rejects on abort(reason) with the reason; never settled by abandonment
+  or GC. close() is single-use (second close rejects, digest unaffected);
+  writes after close reject without disturbing the digest.
+- **`Symbol.dispose`:** errors the digest ("The DigestStream was
+  disposed.") without touching stream state — the stream stays unlocked,
+  zero-length writes still resolve, non-empty writes reject; idempotent;
+  a no-op after close. **Rejection reporting diverges (ledger #2):** the
+  TypeScript implementation marks the digest promise handled, so an
+  abandoned rejected digest produces NO unhandledrejection report; C++
+  reports it. A derived promise (`.then()` with no catch) still reports in
+  both.
+- **Pipes:** a valid pipeTo destination from user streams, transforms, and
+  Response bodies (brand checks pass in both implementations).
+- **Writer accounting:** desiredSize counts the in-flight chunk against the
+  default HWM of 1 and recovers on settlement in BOTH implementations
+  (contrast the compression suite's inert C++ desiredSize); ready stays
+  settled. Write resolutions carry undefined (no thenable check); the
+  digest ArrayBuffer gets exactly one thenable check per ledger #3. The
+  constructor's options-bag getter may re-enter the API safely.
+
+## Compatibility flags
+
+| Flag (enable date) | Selects | Unflagged behavior tested by |
+| --- | --- | --- |
+| `workers_api_getters_setters_on_prototype` (2022-01-31) | `digest` as prototype accessor | `legacyDigestIsOwnInstanceProperty` |
+| `set_tostring_tag` (2024-09-26) | `[object DigestStream]` branding | `legacyToStringTag` |
+| `streams_enable_constructors` + `transformstream_enable_standard_constructor` (2022-11-30) | pipe tests build standard sources | — |
+
+Unlike identity/compression, DigestStream's sink converts exceptions to
+rejected promises directly, so `capture_async_api_throws` has no effect and
+is not pinned.
+
+`digest-cpp-pedantic.wd-test` runs the full module set with the dateless
+opt-in `pedantic_wpt` added, pinning the ABSENCE of pedantic effects
+(the implementation consults the flag nowhere; the reachable
+standard-streams machinery changes nothing the suite pins).
+
+## Divergence ledger (C++ vs TypeScript)
+
+| # | Area | C++ | TypeScript | Pinned in |
+| --- | --- | --- | --- | --- |
+| 1 | Constructor static inheritance | `getPrototypeOf(crypto.DigestStream) !== WritableStream` | `=== WritableStream` | `isRealWritableStreamSubclass` |
+| 2 | Abandoned digest rejection reporting | reported | marked handled, not reported | `abandonedDigestReporting` |
+| 3 | Thenable-check stage for the digest ArrayBuffer (fires exactly once in both) | when the digest promise is awaited | inside close(), as the deferred resolves | `thenInterceptionDuringDigestResolution` |
+
+## Assertion catalogue
+
+| Module | Asserts |
+| --- | --- |
+| `api-surface.js` | WritableStream subclassing incl. ledger #1; branding; brand-checked accessors + dispose; subclassable |
+| `construction.js` | case-insensitive algorithm matching (digest-compared); CRC exact spelling; unknown/missing algorithms throw synchronously; `{name}` object form; all 8 algorithms with sizes; option-bag Web IDL rules with exact TypeError; type-check-before-lookup ordering |
+| `digest-vectors.js` | pinned md5/SHA-256/crc32 outputs for bytes, strings, non-Uint8 views, offset subarrays; AWS SDK checksum vectors; mixed-type write accumulation with bytesWritten; shared digestOf helper |
+| `string-encoding.js` | WTF-8-not-TextEncoder default; toWellFormed matches TextEncoder for every lone-surrogate shape; inert for valid input and byte chunks; identical bytesWritten; every falsy spelling defaults off; ToBoolean coercion; stateful pair-joining across chunks; dangling-lead flush; per-chunk default encoding |
+| `chunk-types.js` | non-byte chunk rejection with exact message; bare-SAB rejection + SAB-view acceptance; DataView offset honored |
+| `digest-promise.js` | promise identity stability across settling; bytesWritten bigint semantics incl. zero-length and UTF-8 counting |
+| `lifecycle.js` | close resolves; abort rejects (incl. after writes); write-after-close rejects without disturbing the digest; double close safe; abandoned stream/writer safe; unused stream safe |
+| `dispose.js` | dispose errors digest + non-empty writes while stream state untouched; zero-length writes still resolve; idempotent; no-op after close |
+| `unhandled-rejection.js` | ledger #2 reporting matrix + derived-promise reporting |
+| `buffer-lifecycle.js` | consume-at-write: post-write mutation/detach invisible; lying metadata getters never consulted |
+| `pipe-integration.js` | pipeTo from user streams; TransformStream chain; Response body |
+| `large-payload.js` | 1MB+ chunk digesting |
+| `gc-interplay.js` | GC never settles an abandoned digest; writer remains operable across GC |
+| `reentrancy.js` | staged thenable-check matrix (ledger #3); write issued from a write continuation preserves accumulation order; options-bag getter re-entering the constructor is safe and its value is honored |
+| `backpressure.js` | desiredSize counts and recovers (parity); ready settled |
+| `legacy-shape.js` | unflagged era: `[object Object]`; digest as own instance property (no prototype accessor); bytesWritten stays a bigint prototype accessor; flow unchanged |
+| `which-impl.js` | implementation detection |

src/tests/streams/digest/BUILD.bazel

@@ -0,0 +1,34 @@
+load("//:build/wd_test.bzl", "wd_test")
+
+# The digest streams suite runs the same test modules under two configs:
+# digest-cpp against the C++ implementation, digest-ts against the
+# TypeScript implementation.
+
+digest_suite_srcs = glob(["*.js"]) + ["digest-modules.capnp"]
+
+wd_test(
+    src = "digest-cpp.wd-test",
+    data = digest_suite_srcs,
+)
+
+wd_test(
+    src = "digest-ts.wd-test",
+    args = ["--experimental"],
+    data = digest_suite_srcs,
+)
+
+# The main C++ cell plus the dateless opt-in pedantic_wpt flag, enforcing
+# that pedantic mode changes nothing on this suite's surface.
+wd_test(
+    src = "digest-cpp-pedantic.wd-test",
+    data = digest_suite_srcs,
+)
+
+# Legacy (pre-flag) regression guard, C++ implementation only. The
+# @all-compat-flags variant is disabled: at the 2999-12-31 date the flags
+# the cell deliberately omits would all turn on.
+wd_test(
+    src = "digest-cpp-legacy.wd-test",
+    data = digest_suite_srcs,
+    generate_all_compat_flags_variant = False,
+)

src/tests/streams/digest/api-surface.js

@@ -0,0 +1,81 @@
+// Copyright (c) 2026 Cloudflare, Inc.
+// Licensed under the Apache 2.0 license found in the LICENSE file or at:
+//     https://opensource.org/licenses/Apache-2.0
+
+// Object shape of crypto.DigestStream: a subclass of the global
+// WritableStream in both implementations, with brand-checked accessors.
+// Divergence: TypeScript uses a genuine `class extends`, so the
+// CONSTRUCTOR's prototype is the WritableStream function itself; the C++
+// jsg inheritance wires the instance prototype chain but not the static
+// constructor chain.
+
+import { strictEqual, notStrictEqual, ok, throws } from 'node:assert';
+import { usingTsImpl } from 'which-impl';
+
+export const isRealWritableStreamSubclass = {
+  test() {
+    const stream = new crypto.DigestStream('md5');
+    ok(stream instanceof WritableStream);
+    strictEqual(
+      Object.getPrototypeOf(crypto.DigestStream.prototype),
+      WritableStream.prototype
+    );
+    if (usingTsImpl) {
+      strictEqual(Object.getPrototypeOf(crypto.DigestStream), WritableStream);
+    } else {
+      notStrictEqual(
+        Object.getPrototypeOf(crypto.DigestStream),
+        WritableStream
+      );
+    }
+    // Inherited members must be reachable, not shadowed.
+    strictEqual(typeof stream.getWriter, 'function');
+    strictEqual(stream.locked, false);
+  },
+};
+
+export const toStringTagBranding = {
+  test() {
+    strictEqual(
+      Object.prototype.toString.call(new crypto.DigestStream('md5')),
+      '[object DigestStream]'
+    );
+  },
+};
+
+export const accessorsAreBrandChecked = {
+  test() {
+    // Brand checks, not instanceof checks: a plain object with the right
+    // prototype is still rejected.
+    const fake = Object.create(crypto.DigestStream.prototype);
+    throws(() => fake.digest, { name: 'TypeError' });
+    throws(() => fake.bytesWritten, { name: 'TypeError' });
+    throws(() => fake[Symbol.dispose](), { name: 'TypeError' });
+
+    const desc = Object.getOwnPropertyDescriptor(
+      crypto.DigestStream.prototype,
+      'digest'
+    );
+    strictEqual(typeof desc.get, 'function');
+    strictEqual(desc.set, undefined);
+  },
+};
+
+export const constructorIsSubclassable = {
+  async test() {
+    class MyDigest extends crypto.DigestStream {
+      constructor() {
+        super('md5');
+        this.tag = 'mine';
+      }
+    }
+    const stream = new MyDigest();
+    strictEqual(stream.tag, 'mine');
+    ok(stream instanceof crypto.DigestStream);
+    ok(stream instanceof WritableStream);
+    const writer = stream.getWriter();
+    await writer.write('hello');
+    await writer.close();
+    strictEqual((await stream.digest).byteLength, 16);
+  },
+};

src/tests/streams/digest/backpressure.js

@@ -0,0 +1,25 @@
+// Copyright (c) 2026 Cloudflare, Inc.
+// Licensed under the Apache 2.0 license found in the LICENSE file or at:
+//     https://opensource.org/licenses/Apache-2.0
+
+// Writer accounting under the synchronous hash sink: desiredSize counts
+// the in-flight chunk against the default HWM of 1 and recovers once the
+// write settles, in BOTH implementations (contrast the compression
+// suite, where the C++ writer's desiredSize is inert). ready stays
+// settled — the eager sink never sustains backpressure.
+
+import { strictEqual } from 'node:assert';
+
+export const desiredSizeCountsAndRecovers = {
+  async test() {
+    const stream = new crypto.DigestStream('md5');
+    const writer = stream.getWriter();
+    strictEqual(writer.desiredSize, 1);
+    const writePromise = writer.write(new Uint8Array(100_000));
+    strictEqual(writer.desiredSize, 0);
+    await writePromise;
+    strictEqual(writer.desiredSize, 1);
+    await writer.ready;
+    await writer.close();
+  },
+};

src/tests/streams/digest/buffer-lifecycle.js

@@ -0,0 +1,63 @@
+// Copyright (c) 2026 Cloudflare, Inc.
+// Licensed under the Apache 2.0 license found in the LICENSE file or at:
+//     https://opensource.org/licenses/Apache-2.0
+
+// Input buffer lifecycle: the hash update runs inside write(), so the
+// chunk's bytes are consumed before the write settles — mutation after the
+// write cannot change the digest, and the chunk is never retained.
+
+import { deepStrictEqual } from 'node:assert';
+import { digestOf } from 'digest-vectors';
+
+export const mutationAfterWriteIsInvisible = {
+  async test() {
+    const buf = new Uint8Array([1, 2, 3, 4]);
+    const stream = new crypto.DigestStream('crc32');
+    const writer = stream.getWriter();
+    await writer.write(buf);
+    buf.fill(0xff);
+    await writer.close();
+    deepStrictEqual(
+      new Uint8Array(await stream.digest),
+      await digestOf('crc32', new Uint8Array([1, 2, 3, 4]))
+    );
+  },
+};
+
+export const detachAfterWriteIsInvisible = {
+  async test() {
+    const buf = new Uint8Array([1, 2, 3, 4]);
+    const stream = new crypto.DigestStream('crc32');
+    const writer = stream.getWriter();
+    await writer.write(buf);
+    buf.buffer.transfer();
+    await writer.close();
+    deepStrictEqual(
+      new Uint8Array(await stream.digest),
+      await digestOf('crc32', new Uint8Array([1, 2, 3, 4]))
+    );
+  },
+};
+
+export const lyingMetadataNeverConsulted = {
+  async test() {
+    // Buffer metadata comes from internal slots; shadowing own getters are
+    // never invoked.
+    const view = new TextEncoder().encode('real');
+    for (const key of ['byteLength', 'byteOffset', 'buffer']) {
+      Object.defineProperty(view, key, {
+        get() {
+          throw new Error(`${key} getter must not be called`);
+        },
+      });
+    }
+    const stream = new crypto.DigestStream('md5');
+    const writer = stream.getWriter();
+    await writer.write(view);
+    await writer.close();
+    deepStrictEqual(
+      new Uint8Array(await stream.digest),
+      await digestOf('md5', 'real')
+    );
+  },
+};