Skip to content

worker-shell: redirected cat output and heredocs store non-ASCII bytes double-encoded (é becomes c3 83 c2 a9) #286

Open

Description

@BenKnill

Describe the bug

In the worker-shell backend, a file written by redirecting cat output stores its non-ASCII bytes UTF-8-encoded a second time. A heredoc containing é (c3 a9) is stored as c3 83 c2 a9. The command exits 0 with empty stderr, so nothing signals the corruption.

Command run in the worker shell Stored bytes Expected
cat > o.txt <<'EOF' / é / EOF c3 83 c2 a9 0a c3 a9 0a
printf 'a\n' > o.txt; cat >> o.txt <<'EOF' … 61 0a c3 83 c2 a9 0a 61 0a c3 a9 0a
cat a.txt > b.txt (a.txt holds é\n) c3 83 c2 a9 0a c3 a9 0a
head -n1 a.txt > o.txt c3 83 c2 a9 0a c3 a9 0a
cat a.txt | tee o.txt c3 83 c2 a9 0a c3 a9 0a
echo w4kK | base64 -d > o.txt c3 83 c2 89 0a c3 89 0a
printf 'é\n' > o.txt (control) c3 a9 0a c3 a9 0a

echo, printf, sed, sort and the cp command store the correct bytes.

Expected behavior

The file holds the bytes the command wrote, as it does in GNU bash and in just-bash over its own InMemoryFs.

Steps to reproduce

Run this through WorkerShellBackend:

cat > o.txt <<'EOF'
é
EOF
wc -c < o.txt

GNU bash prints 3. The worker shell prints 5, and the host's workspace.fs.readFile("/workspace/o.txt") returns the bytes c3 83 c2 a9 0a.

Cause

just-bash passes redirected command output to IFileSystem.writeFile and appendFile as a byte string, one char per byte, with the encoding argument "binary". cat and heredocs produce this byte output, and tee writes the same way. WorkspaceFsAdapter.writeFile and appendFile ignore that argument (adapter.ts L152-L172). writeFile hands the string to fs.writeFile, which encodes it as UTF-8, and appendFile runs it through TextEncoder, so every byte at or above 0x80 becomes two bytes. printf and echo produce text, which just-bash tags "utf8", so they are unaffected.

Possible fix

Honour "binary" and "latin1" in both methods and store such strings as bytes. The patch below, against 071b2b2, adds two tests to adapter.test.ts; both fail before the change. With it:

  • the cases in the table store the expected bytes, the same as just-bash over InMemoryFs;
  • adapter.test.ts passes (34 tests), and the package's default vitest suite passes (99 files, 1,710 tests);
  • the other npm test configs pass: worker-backend (7 tests), proxy (12), script-runner (78) and stub-soak (5);
  • npm run typecheck passes, and oxlint and oxfmt --check report nothing for the two files.
Patch (adapter.ts +31/−4, adapter.test.ts +22)
diff --git a/packages/computer/src/backends/worker-shell/adapter.test.ts b/packages/computer/src/backends/worker-shell/adapter.test.ts
index 2de3488..f5b3f1d 100644
--- a/packages/computer/src/backends/worker-shell/adapter.test.ts
+++ b/packages/computer/src/backends/worker-shell/adapter.test.ts
@@ -236,6 +236,28 @@ describe("WorkspaceFsAdapter — writes", () => {
     expect(await adapter.readlink("/link")).toBe("/target");
   });
 
+  it("writeFile and appendFile store a binary string as one byte per char", async () => {
+    // just-bash hands shell output to the filesystem as a byte string
+    // tagged "binary": "\xc3\xa9" is the UTF-8 encoding of é.
+    await adapter.writeFile("/b", "\xc3\xa9", "binary");
+    await adapter.appendFile("/b", "\xff\n", { encoding: "latin1" });
+    const back = new Uint8Array(
+      await new Response(await workspace.fs.readFile("/b")).arrayBuffer(),
+    );
+    expect(Array.from(back)).toEqual([0xc3, 0xa9, 0xff, 0x0a]);
+  });
+
+  it("keeps non-ASCII bytes when the shell redirects cat, a heredoc or tee", async () => {
+    const bash = new Bash({ fs: adapter as never, cwd: "/" });
+    const result = await bash.exec(
+      "cat > /h <<'EOF'\né\nEOF\ncat /h > /c; cat /h >> /a; cat /h | tee /t >/dev/null",
+    );
+    expect(result).toMatchObject({ exitCode: 0, stderr: "" });
+    for (const path of ["/h", "/c", "/a", "/t"]) {
+      expect(await workspace.fs.readFile(path, "utf8")).toBe("é\n");
+    }
+  });
+
   it("discards shell redirection writes to /dev/null", async () => {
     const bash = new Bash({ fs: adapter as never, cwd: "/" });
     const result = await bash.exec("printf hello >/dev/null && cat /dev/null");
diff --git a/packages/computer/src/backends/worker-shell/adapter.ts b/packages/computer/src/backends/worker-shell/adapter.ts
index 2d857fb..716fadc 100644
--- a/packages/computer/src/backends/worker-shell/adapter.ts
+++ b/packages/computer/src/backends/worker-shell/adapter.ts
@@ -74,6 +74,7 @@ interface DirentEntry {
 }
 
 type AdapterReadFileOptions = { encoding?: "utf8" | null } | "utf8" | null | undefined;
+type AdapterWriteFileOptions = { encoding?: string | null } | string | null | undefined;
 
 export class WorkspaceFsAdapter {
   readonly #fs: WorkspaceFs;
@@ -149,12 +150,20 @@ export class WorkspaceFsAdapter {
 
   // --- Writes ------------------------------------------------------
 
-  async writeFile(path: string, content: string | Uint8Array, _options?: unknown): Promise<void> {
+  async writeFile(
+    path: string,
+    content: string | Uint8Array,
+    options?: AdapterWriteFileOptions,
+  ): Promise<void> {
     if (isDevNull(path)) return;
-    await this.#fs.writeFile(path, content);
+    await this.#fs.writeFile(path, binaryContent(content, options));
   }
 
-  async appendFile(path: string, content: string | Uint8Array, _options?: unknown): Promise<void> {
+  async appendFile(
+    path: string,
+    content: string | Uint8Array,
+    options?: AdapterWriteFileOptions,
+  ): Promise<void> {
     if (isDevNull(path)) return;
     let existing: Uint8Array;
     try {
@@ -164,7 +173,8 @@ export class WorkspaceFsAdapter {
       if ((err as { code?: string }).code !== "ENOENT") throw err;
       existing = new Uint8Array(0);
     }
-    const addition = typeof content === "string" ? new TextEncoder().encode(content) : content;
+    const data = binaryContent(content, options);
+    const addition = typeof data === "string" ? new TextEncoder().encode(data) : data;
     const out = new Uint8Array(existing.byteLength + addition.byteLength);
     out.set(existing, 0);
     out.set(addition, existing.byteLength);
@@ -280,6 +290,23 @@ function missingPath(path: string): Error & { code: string; path: string } {
   return error;
 }
 
+// just-bash writes shell output (redirections, tee) as a byte string
+// tagged "binary": each char is one byte, so UTF-8 text arrives
+// already encoded. Pass those bytes through; only an untagged or
+// "utf8" string is text for the store to encode.
+function binaryContent(
+  content: string | Uint8Array,
+  options: AdapterWriteFileOptions,
+): string | Uint8Array {
+  const encoding = typeof options === "string" ? options : options?.encoding;
+  if (typeof content !== "string" || (encoding !== "binary" && encoding !== "latin1")) {
+    return content;
+  }
+  const bytes = new Uint8Array(content.length);
+  for (let i = 0; i < content.length; i++) bytes[i] = content.charCodeAt(i);
+  return bytes;
+}
+
 function mapStat(s: WorkspaceStatResult): FsStat {
   return {
     isFile: s.isFile,

Environment

  • @cloudflare/computer 0.8.0 and main at 071b2b2. The adapter's write path is the same in both.
  • just-bash 3.2.0 (the lockfile version) and 3.6.0 (latest on npm) give the same result.
  • It was reproduced in two ways, on macOS with Node 26.12.0:
    • In Node: new Bash({ fs: new WorkspaceFsAdapter(stub) }) over an in-process Workspace with SQLiteTestStorage, as in adapter.test.ts. This covered every row of the table.
    • In local workerd 1.20261006.1, through @cloudflare/vitest-pool-workers 0.22.0 and tests/wrangler.worker-backend.jsonc: /exec, then /read of the Durable Object's SQLite workspace. The heredoc write and append rows were corrupted, wc -c printed 5, and the printf control was correct. With the patch and a rebuilt shell bundle, all were correct.
  • It has not been tested on a deployed Worker.

This report and the patch were prepared with AI assistance (Claude Opus 5.5).

Activity

  1. added a commit that references this issue on Oct 29, 2026
    3d76f6a
  2. github-actions commented on Oct 29, 2026

    @github-actions
    Contributor

    I've opened a fix: #287

    • Cause: WorkspaceFsAdapter.writeFile and appendFile ignored the "binary" encoding just-bash uses for cat, heredoc and tee output, so the store UTF-8-encoded already-encoded bytes a second time.
    • Change: strings tagged "binary" or "latin1" are now converted to one byte per char before being stored. I confirmed the new tests fail on main and pass with the change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingpriority: highHigh impact: fix soon

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions