experimental_tokio on Emscripten: cancellation, crash-recovery and request-lifetime gaps found while testing #1111

I've been testing experimental_tokio (worker-build --emscripten) with real crates: hyper, redis, tokio-postgres, sqlx 0.8 and Durable Object pools. The success paths work well, and the same app gives byte-identical output natively and as a Worker. I then ran 35 failure-mode tests, each against a native-Tokio control, and a few behaviours look like gaps in how the Tokio loop is tied to the Worker's lifecycle rather than bugs in one place. I'd like to check whether they're known or intended before proposing changes.

Setup: worker 0.8.7 from main @1d668cf; tokio guybedford/tokio tag 1.53.1-cf.emscripten; mio 1.2.3-cf.emscripten; wasm-bindgen 0.2.129; emsdk 6.0.10 plus the worker-build patches; rustc 1.100.0-beta.7. Run with local wrangler dev 4.151.2 (workerd 1.20261019.1) only, so production may differ. Each case below has raw logs; the write loss (3) and the DNS hang (6) reproduced 3 of 3 times.

1. A request that workerd cancels never drops its Tokio loop, which leaks fds

If workerd cancels a request ("code had hung"), the root future never settles. Its isolated loop is never dropped, so its sockets are never closed. Each cancelled request leaked 4 wasm fds (a fresh fd number went 34 → 114 → 194 → 994 over 240 cancelled requests), and they were never reclaimed. Later, the stale epoll listeners threw Disallowed operation called within global scope (10 times).

Client-side aborts (curl --max-time 0.2 × 500 on a 10 s sleep) did not leak locally, because those requests ran to completion.

As far as I can tell, nothing connects workerd's cancellation to dropping the root future in schedule_isolated. Is a cancellation hook (IoContext abort → drop the root) planned?

2. A panic inside the loop's drive leaves the isolate broken

struct PanicWaker;
impl std::task::Wake for PanicWaker {
    fn wake(self: Arc<Self>) { panic!("boom") }
}
// poll a tokio::time::sleep(20ms) once with Waker::from(Arc::new(PanicWaker)),
// then sleep 100ms so the timer driver calls wake() during its turn

The panic aborts the module (Aborted(), then RuntimeError: unreachable), but the same isolate keeps receiving requests. Synchronous routes and JS Delay still work, while every Tokio-driven wait (timer, socket, DO pool) hangs until "code had hung". Natively the panic is caught. The std-target build has abort handling and reinit, but shim-emscripten.js doesn't, and callUserCallback returns early once ABORT is set. Should the Emscripten path get the same reinit (and DO recreation) as the std target?

3. Bytes accepted by write() are lost when the request ends before the peer reads them

When the peer stops reading, the client writes until write() blocks (589,824 bytes accepted) and then returns its response. The peer received 539,996 bytes, so 49,828 bytes were lost, in 3 of 3 runs. Natively all 745,004 bytes arrive after the drop. If the request stays alive 7 s longer, all 589,824 bytes arrive. Request teardown seems to discard what workerd had already accepted from the socket.

That's consistent with request-scoped I/O, but surprising for POSIX-style code, where a successful write plus close means the kernel delivers the data. Related, and probably expected:

• tasks spawned in a handler are dropped when it returns;
• ctx.wait_until(tokio::time::sleep(..)) panics with "no reactor running".

Is the write loss intended? If it is, documenting it, or offering a "flush before responding" helper, would help.

4. Dropping a Fetch::send future doesn't abort the upstream fetch

With timeout(100ms) around Fetch::send() on a 5 s upstream, the upstream connection stayed open until the handler returned (+3.0 s). With send_with_signal + abort() it closed after 2 ms, and with tokio::net after 1 ms. An abort-on-drop guard (like reqwest's wasm backend has) would make Fetch behave like other Rust futures under timeout/select!.

5. Connection errors lose their kind

TcpStream::connect to a closed port gives os error 29 (EIO, kind InputOutputError) in 1–6 ms; native gives ConnectionRefused. A connect timeout also ends in EIO (after 75 s); native gives TimedOut. The cause: workerd's socket errors carry no .code, so the Emscripten glue's nodeErrToErrno falls back to EIO. Should this be fixed in workerd (set code on the error) or in the glue?

A related case: a TCP reset reads as a clean EOF (Ok(n) then Ok(0); only take_error() shows EIO), where native gives ConnectionReset. emscripten-core/emscripten#27724 already contains the fix. Its test injects the error from JS; I have a test with a real RST peer (SO_LINGER 0) that passes 30/30 with that hunk, and I'm happy to contribute it there.

6. A JS exception thrown from an import during a poll wedges the task

Found while root-causing a hang: tokio::net::lookup_host("www.github.com:443") (a CNAME) never completes, and the timeout(10s) around it never fires.

1. workerd's dns.lookup({all:true}) returns the CNAME target name as an address: [{address:"github.com.",family:4},{address:"20.250.119.67",family:4},{address:"github.com.",family:6}]. (workerd fix to follow.)
2. The glue keeps the unparseable entry. Fix: NODERAWSOCKETS: skip unparsable addresses in getaddrinfo results emscripten-core/emscripten#28049.
3. The patched emscripten_dns_lookup_result then throws TypeError: Cannot read properties of null (reading '0') inside allocaddrinfo, synchronously, from the import called at net/emscripten_dns.rs:45.

That exception unwinds through the Rust frames (exnref), so drops run and the timeout's sleep is dropped too. Then it is swallowed by the promise-callback wrapper around drive_microtask (it surfaces only as unhandledrejection: 0). Neither the task harness's catch_unwind nor the drive's catches it. There's no abort and no JoinError.

The root task's poll never returns, so it stays RUNNING forever: the request hangs until "code had hung", and the isolate keeps serving. The DNS case is just the trigger: any JS import that throws during a drive should behave this way. Should a foreign exception during a drive abort the loop, or fail the task loudly?

Two smaller things in the same path:

• the worker-build patch's emscripten_dns_lookup_result returns -3 for both "pending" and EAI_AGAIN-style failures. With a forced -3 failure, emscripten_dns.rs:53 calls clear_ready and waits forever. Once readiness has fired, -3 can't mean "pending", so treating it as an error would be a plain fix in the fork. The Add _promise variants for __async library functions emscripten-core/emscripten#27880 / Make getaddrinfo an asynchronous lookup with promise and callback variants emscripten-core/emscripten#27881 redesign removes the ambiguity, but it needs emscripten_dns.rs re-ported to its callback API.
• a try/catch around writeAddrInfo in that patched function would turn any future throw into an error code.

7. Open question: the ambient loop is per isolate, not per Durable Object

The ambient loop is a thread-local shared by every DO in the isolate. The macro comment says "a Durable Object is one I/O context". Locally, two DOs (id_from_name("a") / ("b")) in one isolate ran 92 of 92 concurrent requests with no cross-context errors, so this may be fine. Is production different?

I have a small test crate (Worker plus native control running the same test bodies) and raw logs for each case, and I'm happy to share or turn any of them into tests here. If a separate issue per item is easier to track, I can split this up.