Skip to content

Commit 67f249f

Browse files
rcccfrcccf
committed
Introduce WebRTC -> JPEG Adapter Demo
1 parent 8f87212 commit 67f249f

18 files changed

Lines changed: 12778 additions & 0 deletions

‎video-to-jpeg/.gitignore‎

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
# Logs
2+
3+
logs
4+
_.log
5+
npm-debug.log_
6+
yarn-debug.log*
7+
yarn-error.log*
8+
lerna-debug.log*
9+
.pnpm-debug.log*
10+
11+
# Diagnostic reports (https://nodejs.org/api/report.html)
12+
13+
report.[0-9]_.[0-9]_.[0-9]_.[0-9]_.json
14+
15+
# Runtime data
16+
17+
pids
18+
_.pid
19+
_.seed
20+
\*.pid.lock
21+
22+
# Directory for instrumented libs generated by jscoverage/JSCover
23+
24+
lib-cov
25+
26+
# Coverage directory used by tools like istanbul
27+
28+
coverage
29+
\*.lcov
30+
31+
# nyc test coverage
32+
33+
.nyc_output
34+
35+
# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files)
36+
37+
.grunt
38+
39+
# Bower dependency directory (https://bower.io/)
40+
41+
bower_components
42+
43+
# node-waf configuration
44+
45+
.lock-wscript
46+
47+
# Compiled binary addons (https://nodejs.org/api/addons.html)
48+
49+
build/Release
50+
51+
# Dependency directories
52+
53+
node_modules/
54+
jspm_packages/
55+
.wrangler/
56+
dist/
57+
.DS_Store
58+
public/assets/
59+
web_modules/
60+
61+
# TypeScript cache
62+
63+
\*.tsbuildinfo
64+
65+
# Optional npm cache directory
66+
67+
.npm
68+
69+
# Optional eslint cache
70+
71+
.eslintcache
72+
73+
# Optional stylelint cache
74+
75+
.stylelintcache
76+
77+
# Microbundle cache
78+
79+
.rpt2_cache/
80+
.rts2_cache_cjs/
81+
.rts2_cache_es/
82+
.rts2_cache_umd/
83+
84+
# Optional REPL history
85+
86+
.node_repl_history
87+
88+
# Output of 'npm pack'
89+
90+
\*.tgz
91+
92+
# Yarn Integrity file
93+
94+
.yarn-integrity
95+
96+
# parcel-bundler cache (https://parceljs.org/)
97+
98+
.cache
99+
.parcel-cache
100+
101+
# Next.js build output
102+
103+
.next
104+
out
105+
106+
# Nuxt.js build / generate output
107+
108+
.nuxt
109+
dist
110+
111+
# Gatsby files
112+
113+
.cache/
114+
115+
# Comment in the public line in if your project uses Gatsby and not Next.js
116+
117+
# https://nextjs.org/blog/next-9-1#public-directory-support
118+
119+
# public
120+
121+
# vuepress build output
122+
123+
.vuepress/dist
124+
125+
# vuepress v2.x temp and cache directory
126+
127+
.temp
128+
.cache
129+
130+
# Docusaurus cache and generated files
131+
132+
.docusaurus
133+
134+
# Serverless directories
135+
136+
.serverless/
137+
138+
# FuseBox cache
139+
140+
.fusebox/
141+
142+
# DynamoDB Local files
143+
144+
.dynamodb/
145+
146+
# TernJS port file
147+
148+
.tern-port
149+
150+
# Stores VSCode versions used for testing VSCode extensions
151+
152+
.vscode-test
153+
154+
# yarn v2
155+
156+
.yarn/cache
157+
.yarn/unplugged
158+
.yarn/build-state.yml
159+
.yarn/install-state.gz
160+
.pnp.\*
161+
162+
# wrangler project
163+
164+
.dev.vars*
165+
!.dev.vars.example
166+
.env*
167+
!.env.example
168+
.wrangler/
169+
170+
# Local tooling (emsdk) installed by scripts/bootstrap.sh
171+
.tooling/

‎video-to-jpeg/ARCHITECTURE.md‎

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
# Architecture and Technical Details
2+
3+
This document provides in-depth technical information about the video-to-jpeg demo's architecture, API flows, and implementation details.
4+
5+
## High-level Architecture
6+
7+
### Components
8+
9+
- **Cloudflare Worker** (`src/index.ts`)
10+
- Routes requests based on `/<session>/...`.
11+
- Serves the static HTML UI from `src/player.html`.
12+
- Forwards all `/⟨session⟩/video/*` requests to the `VideoAdapter` Durable Object instance derived from `session`.
13+
14+
- **`VideoAdapter` Durable Object** (`src/video-adapter.ts`)
15+
- Owns the state for a single `session` name:
16+
- Realtime SFU session ID.
17+
- Video track name.
18+
- WebSocket adapter ID.
19+
- HTTP endpoints:
20+
- `POST /⟨session⟩/video/connect` — create SFU session, publish video via `autoDiscover`.
21+
- `POST /⟨session⟩/video/start-forwarding` — configure a WebSocket adapter with `outputCodec: "jpeg"`.
22+
- `POST /⟨session⟩/video/stop-forwarding` — close the adapter (idempotent).
23+
- WebSocket endpoints:
24+
- `WS /⟨session⟩/video/sfu-subscribe` — SFU → DO, receives `Packet` messages containing JPEG payloads.
25+
- `WS /⟨session⟩/video/viewer` — DO → browsers, fans out raw JPEG payloads to viewers.
26+
27+
- **SFU helper** (`src/shared/sfu-utils.ts`)
28+
- Wraps Realtime SFU REST API calls:
29+
- `createSession()`
30+
- `addTracksAutoDiscoverForVideo()`
31+
- `pullTrackToWebSocket()` (with `outputCodec: "jpeg"`)
32+
- `closeWebSocketAdapter()`
33+
- Knows how to decode the `Packet` protobuf and extract JPEG payloads.
34+
35+
- **Frontend app** (`src/web/app.ts`)
36+
- Publisher:
37+
- WebRTC offer/answer negotiation with the SFU via the Worker.
38+
- Controls start/stop of JPEG streaming via adapter endpoints.
39+
- Renders local camera video and snapshots.
40+
- Viewer:
41+
- Connects to the viewer WebSocket and renders the JPEG frames.
42+
43+
## Publisher Workflow (Detailed)
44+
45+
When you click buttons on the **publisher** page, here's what happens under the hood:
46+
47+
### 1. Connect Camera
48+
49+
1. The browser requests camera access via `getUserMedia`.
50+
2. A `RTCPeerConnection` is created with your camera tracks.
51+
3. The frontend sends an SDP offer to the Worker:
52+
- `POST /my-session/video/connect` with `{ sessionDescription: offer }`.
53+
4. The `VideoAdapter` Durable Object:
54+
- Creates an SFU session via `POST /v1/apps/{appId}/sessions/new`.
55+
- Publishes your track via `autoDiscover` onto that session.
56+
- Returns an SFU answer, which the frontend sets as the remote description.
57+
58+
### 2. Start JPEG Stream
59+
60+
1. Frontend calls `POST /my-session/video/start-forwarding`.
61+
2. `VideoAdapter` configures a **WebSocket adapter** with:
62+
- `location: "remote"`
63+
- `sessionId: <publisher-sfu-session-id>`
64+
- `trackName: <video-track-name>`
65+
- `endpoint: wss://.../my-session/video/sfu-subscribe`
66+
- `outputCodec: "jpeg"`
67+
3. The Realtime SFU starts sending JPEG frames (~1 FPS) to the Durable Object over WebSocket.
68+
4. The publisher page opens a viewer WebSocket as well (same as the viewer page) and shows the JPEG snapshots.
69+
70+
### 3. Stop JPEG Stream
71+
72+
- Frontend calls `POST /my-session/video/stop-forwarding`.
73+
- `VideoAdapter` closes the WebSocket adapter via the Realtime SFU API.
74+
75+
## Viewer Workflow (Detailed)
76+
77+
On the **viewer** page (`/my-session/viewer`):
78+
79+
1. The frontend opens a WebSocket to:
80+
```
81+
ws(s)://<host>/my-session/video/viewer
82+
```
83+
84+
2. The `VideoAdapter`:
85+
- Receives each adapter frame as a `Packet` protobuf from the SFU.
86+
- Extracts the JPEG `payload`.
87+
- Broadcasts the JPEG bytes to all connected viewer sockets.
88+
89+
3. The browser receives each message as a binary `Blob`, wraps it in an `ObjectURL`, and assigns it to an `<img>` element.
90+
91+
If a viewer connects late, the `VideoAdapter` sends the **last stored frame** immediately upon connection so the UI shows something even before the next snapshot arrives.
92+
93+
## API Routes Reference
94+
95+
### UI pages
96+
97+
- `GET /⟨session⟩/publisher` — Publisher interface with camera controls
98+
- `GET /⟨session⟩/viewer` — Viewer interface showing JPEG stream
99+
100+
### HTTP endpoints (VideoAdapter)
101+
102+
- `POST /⟨session⟩/video/connect` — Create SFU session and publish camera
103+
- `POST /⟨session⟩/video/start-forwarding` — Start WebSocket adapter with JPEG output
104+
- `POST /⟨session⟩/video/stop-forwarding` — Stop adapter and close connection
105+
106+
### WebSocket endpoints
107+
108+
- `WS /⟨session⟩/video/sfu-subscribe` — SFU → Durable Object (JPEG `Packet` messages)
109+
- `WS /⟨session⟩/video/viewer` — Durable Object → browsers (raw JPEG bytes)
110+
111+
### Debug endpoints
112+
113+
- `DELETE /⟨session⟩` — Calls `VideoAdapter.destroy()` to close all sockets and wipe state for that session. This is unauthenticated for demo purposes; add authentication before using in production.
114+
115+
## Protobuf Message Format
116+
117+
The WebSocket adapter sends video frames as protobuf `Packet` messages:
118+
119+
```proto
120+
syntax = "proto3";
121+
122+
message Packet {
123+
uint32 sequenceNumber = 1; // sequence number (used for audio; may be unset for video)
124+
uint32 timestamp = 2; // timestamp for synchronization
125+
bytes payload = 5; // media payload (PCM audio or JPEG video)
126+
}
127+
```
128+
129+
For JPEG video, the `payload` field contains the raw JPEG image bytes, which can be directly rendered in a browser `<img>` element or processed further.
130+
131+
## Durable Object State Persistence
132+
133+
The `VideoAdapter` persists the following state across requests:
134+
135+
- `sfuSessionId` — The Realtime SFU session ID
136+
- `videoTrackName` — The name of the published video track
137+
- `sfuAdapterId` — The WebSocket adapter ID (if active)
138+
- `sessionName` — Human-readable session identifier
139+
140+
This state is stored in Durable Object storage and survives Worker restarts. The "Reset Session" button calls `DELETE /<session>` to clear this state.
141+
142+
## Video Processing Pipeline
143+
144+
```
145+
Camera (Browser)
146+
↓ WebRTC
147+
Realtime SFU
148+
↓ WebSocket Adapter (outputCodec: "jpeg")
149+
VideoAdapter Durable Object
150+
↓ WebSocket (binary JPEG)
151+
Viewer Browsers
152+
```
153+
154+
The Realtime SFU transcodes the incoming video stream to JPEG at approximately 1 FPS and sends each frame as a protobuf `Packet` to the Durable Object, which broadcasts it to all connected viewers.

0 commit comments

Comments
 (0)