Repository navigation
worker-shell: redirected cat output and heredocs store non-ASCII bytes double-encoded (é becomes c3 83 c2 a9) #286
Description
Describe the bug
In the worker-shell backend, a file written by redirecting cat output stores its non-ASCII bytes UTF-8-encoded a second time. A heredoc containing é (c3 a9) is stored as c3 83 c2 a9. The command exits 0 with empty stderr, so nothing signals the corruption.
| Command run in the worker shell | Stored bytes | Expected |
|---|---|---|
cat > o.txt <<'EOF' / é / EOF |
c3 83 c2 a9 0a |
c3 a9 0a |
printf 'a\n' > o.txt; cat >> o.txt <<'EOF' … |
61 0a c3 83 c2 a9 0a |
61 0a c3 a9 0a |
cat a.txt > b.txt (a.txt holds é\n) |
c3 83 c2 a9 0a |
c3 a9 0a |
head -n1 a.txt > o.txt |
c3 83 c2 a9 0a |
c3 a9 0a |
cat a.txt | tee o.txt |
c3 83 c2 a9 0a |
c3 a9 0a |
echo w4kK | base64 -d > o.txt |
c3 83 c2 89 0a |
c3 89 0a |
printf 'é\n' > o.txt (control) |
c3 a9 0a |
c3 a9 0a |
echo, printf, sed, sort and the cp command store the correct bytes.
Expected behavior
The file holds the bytes the command wrote, as it does in GNU bash and in just-bash over its own InMemoryFs.
Steps to reproduce
Run this through WorkerShellBackend:
cat > o.txt <<'EOF'
é
EOF
wc -c < o.txtGNU bash prints 3. The worker shell prints 5, and the host's workspace.fs.readFile("/workspace/o.txt") returns the bytes c3 83 c2 a9 0a.
Cause
just-bash passes redirected command output to IFileSystem.writeFile and appendFile as a byte string, one char per byte, with the encoding argument "binary". cat and heredocs produce this byte output, and tee writes the same way. WorkspaceFsAdapter.writeFile and appendFile ignore that argument (adapter.ts L152-L172). writeFile hands the string to fs.writeFile, which encodes it as UTF-8, and appendFile runs it through TextEncoder, so every byte at or above 0x80 becomes two bytes. printf and echo produce text, which just-bash tags "utf8", so they are unaffected.
Possible fix
Honour "binary" and "latin1" in both methods and store such strings as bytes. The patch below, against 071b2b2, adds two tests to adapter.test.ts; both fail before the change. With it:
- the cases in the table store the expected bytes, the same as just-bash over
InMemoryFs; adapter.test.tspasses (34 tests), and the package's default vitest suite passes (99 files, 1,710 tests);- the other
npm testconfigs pass: worker-backend (7 tests), proxy (12), script-runner (78) and stub-soak (5); npm run typecheckpasses, and oxlint andoxfmt --checkreport nothing for the two files.
Patch (adapter.ts +31/−4, adapter.test.ts +22)
diff --git a/packages/computer/src/backends/worker-shell/adapter.test.ts b/packages/computer/src/backends/worker-shell/adapter.test.ts
index 2de3488..f5b3f1d 100644
--- a/packages/computer/src/backends/worker-shell/adapter.test.ts
+++ b/packages/computer/src/backends/worker-shell/adapter.test.ts
@@ -236,6 +236,28 @@ describe("WorkspaceFsAdapter — writes", () => {
expect(await adapter.readlink("/link")).toBe("/target");
});
+ it("writeFile and appendFile store a binary string as one byte per char", async () => {
+ // just-bash hands shell output to the filesystem as a byte string
+ // tagged "binary": "\xc3\xa9" is the UTF-8 encoding of é.
+ await adapter.writeFile("/b", "\xc3\xa9", "binary");
+ await adapter.appendFile("/b", "\xff\n", { encoding: "latin1" });
+ const back = new Uint8Array(
+ await new Response(await workspace.fs.readFile("/b")).arrayBuffer(),
+ );
+ expect(Array.from(back)).toEqual([0xc3, 0xa9, 0xff, 0x0a]);
+ });
+
+ it("keeps non-ASCII bytes when the shell redirects cat, a heredoc or tee", async () => {
+ const bash = new Bash({ fs: adapter as never, cwd: "/" });
+ const result = await bash.exec(
+ "cat > /h <<'EOF'\né\nEOF\ncat /h > /c; cat /h >> /a; cat /h | tee /t >/dev/null",
+ );
+ expect(result).toMatchObject({ exitCode: 0, stderr: "" });
+ for (const path of ["/h", "/c", "/a", "/t"]) {
+ expect(await workspace.fs.readFile(path, "utf8")).toBe("é\n");
+ }
+ });
+
it("discards shell redirection writes to /dev/null", async () => {
const bash = new Bash({ fs: adapter as never, cwd: "/" });
const result = await bash.exec("printf hello >/dev/null && cat /dev/null");
diff --git a/packages/computer/src/backends/worker-shell/adapter.ts b/packages/computer/src/backends/worker-shell/adapter.ts
index 2d857fb..716fadc 100644
--- a/packages/computer/src/backends/worker-shell/adapter.ts
+++ b/packages/computer/src/backends/worker-shell/adapter.ts
@@ -74,6 +74,7 @@ interface DirentEntry {
}
type AdapterReadFileOptions = { encoding?: "utf8" | null } | "utf8" | null | undefined;
+type AdapterWriteFileOptions = { encoding?: string | null } | string | null | undefined;
export class WorkspaceFsAdapter {
readonly #fs: WorkspaceFs;
@@ -149,12 +150,20 @@ export class WorkspaceFsAdapter {
// --- Writes ------------------------------------------------------
- async writeFile(path: string, content: string | Uint8Array, _options?: unknown): Promise<void> {
+ async writeFile(
+ path: string,
+ content: string | Uint8Array,
+ options?: AdapterWriteFileOptions,
+ ): Promise<void> {
if (isDevNull(path)) return;
- await this.#fs.writeFile(path, content);
+ await this.#fs.writeFile(path, binaryContent(content, options));
}
- async appendFile(path: string, content: string | Uint8Array, _options?: unknown): Promise<void> {
+ async appendFile(
+ path: string,
+ content: string | Uint8Array,
+ options?: AdapterWriteFileOptions,
+ ): Promise<void> {
if (isDevNull(path)) return;
let existing: Uint8Array;
try {
@@ -164,7 +173,8 @@ export class WorkspaceFsAdapter {
if ((err as { code?: string }).code !== "ENOENT") throw err;
existing = new Uint8Array(0);
}
- const addition = typeof content === "string" ? new TextEncoder().encode(content) : content;
+ const data = binaryContent(content, options);
+ const addition = typeof data === "string" ? new TextEncoder().encode(data) : data;
const out = new Uint8Array(existing.byteLength + addition.byteLength);
out.set(existing, 0);
out.set(addition, existing.byteLength);
@@ -280,6 +290,23 @@ function missingPath(path: string): Error & { code: string; path: string } {
return error;
}
+// just-bash writes shell output (redirections, tee) as a byte string
+// tagged "binary": each char is one byte, so UTF-8 text arrives
+// already encoded. Pass those bytes through; only an untagged or
+// "utf8" string is text for the store to encode.
+function binaryContent(
+ content: string | Uint8Array,
+ options: AdapterWriteFileOptions,
+): string | Uint8Array {
+ const encoding = typeof options === "string" ? options : options?.encoding;
+ if (typeof content !== "string" || (encoding !== "binary" && encoding !== "latin1")) {
+ return content;
+ }
+ const bytes = new Uint8Array(content.length);
+ for (let i = 0; i < content.length; i++) bytes[i] = content.charCodeAt(i);
+ return bytes;
+}
+
function mapStat(s: WorkspaceStatResult): FsStat {
return {
isFile: s.isFile,Environment
@cloudflare/computer0.8.0 andmainat071b2b2. The adapter's write path is the same in both.- just-bash 3.2.0 (the lockfile version) and 3.6.0 (latest on npm) give the same result.
- It was reproduced in two ways, on macOS with Node 26.12.0:
- In Node:
new Bash({ fs: new WorkspaceFsAdapter(stub) })over an in-processWorkspacewithSQLiteTestStorage, as inadapter.test.ts. This covered every row of the table. - In local workerd 1.20261006.1, through
@cloudflare/vitest-pool-workers0.22.0 andtests/wrangler.worker-backend.jsonc:/exec, then/readof the Durable Object's SQLite workspace. The heredoc write and append rows were corrupted,wc -cprinted5, and theprintfcontrol was correct. With the patch and a rebuilt shell bundle, all were correct.
- In Node:
- It has not been tested on a deployed Worker.
This report and the patch were prepared with AI assistance (Claude Opus 5.5).
Activity
fix(worker-shell): store binary-tagged writes as raw bytes
github-actions commentedon Oct 29, 2026
I've opened a fix: #287
WorkspaceFsAdapter.writeFileandappendFileignored the"binary"encoding just-bash uses forcat, heredoc andteeoutput, so the store UTF-8-encoded already-encoded bytes a second time."binary"or"latin1"are now converted to one byte per char before being stored. I confirmed the new tests fail onmainand pass with the change.