Skip to content

Deterministic Testing

The test suite has no framework dependency and no network. Custom SceneTree runners under tests/ run with godot --headless --script and produce deterministic results:

  • Byte-pinned fixtures. Codec tests compare exact bytes against fixture files.
  • Injected clocks. Reconnect backoff tests step time manually. No sleeps, no wall-clock waiting.
  • Synchronous fake transport. SFFakeTransport is an in-memory test double with injectors (inject_open(), inject_text(), inject_server_message(), inject_binary(), inject_close(), inject_failure()) and recorded sent_text / sent_binary buffers.

Suites

  • Protocol fixtures. The upstream v2 samples (server v0.9.2) are vendored byte-identically under tests/fixtures/upstream/, and the codec is pinned to them. Hand-built fixtures cover all 24 server message variants plus malformed input.
  • Transport. The WebSocketPeer adapter runs against fakes for connect, receive, send, close, error, and backpressure behavior.
  • Client. Connect, authenticate, join, game data, authority, spectators, reconnection, and the WebRTC mesh all run on fake transports and injected clocks.

Run locally

Bash
python3 -E scripts/run-runtime-checks.py all

all runs the private-helper guard, GDScript and Python format/lint checks, strict Python typing, the Prettier and markdownlint checks, and the Godot suites. Subcommands for targeted runs:

  • static: private-helper guard, format/lint, and Python types. No Godot install.
  • python-types: Python format, lint, and strict types.
  • private-helpers: the private-helper static guard.
  • format: gdformat checks.
  • lint: gdlint.
  • prettier: the Prettier half of CI's Source formatting job. Needs npm ci --ignore-scripts first.
  • markdownlint: the markdownlint job of CI's Docs Validation. Same npm ci requirement. To autofix, run python3 scripts/check-source.py markdownlint write. The pin matches the version bundled by the markdownlint-cli2-action in docs-validation.yml - when bumping that action, re-pin package.json to its bundled version.
  • godot: the SceneTree suites.

Smoke (opt-in)

Bash
python3 -E scripts/run-runtime-checks.py smoke

smoke is never part of all. It drives a real WebSocketPeer round-trip against a local RFC 6455 test server: open, echo round-trips, close handshakes in both directions, and refused-dial failure.

Type strictness

Every GDScript warning class the engine matrix registers is pinned to error level in project.godot except three: := type inference and discarded return values stay ignorable by style, and the 4.7-only missing_await stays unpinned because a marker naming it is a parse error on engines that lack the class. The checks apply to the addon as well: the engine default exclude_addons is explicitly disabled, so a warning regression fails CI anywhere in the repo. The small set of statements that are not yet fully typed carry an explicit @warning_ignore marker naming exactly what they trigger: grep for @warning_ignore in addons/ for the live list.

Live server acceptance (opt-in)

Join-only room admission

With an isolated server v0.10.0 or later, run:

Bash
godot --headless --path . --script tests/smoke/run_join_only_acceptance.gd -- ws://127.0.0.1:3536

For v2 and v3, two join-only requests must each return ROOM_NOT_FOUND for the same absent room. An explicit false then creates that room. An explicit true joins it, and both players start a game and exchange exact gameplay. The deterministic suites also reject invalid flags and a missing room code.

Spectator and session acceptance

tests/smoke/run_live_server_acceptance.gd drives real Godot clients through SFWebSocketTransport. Use an isolated local server. The drill creates rooms and changes their membership. It requires a server with spectator_relay and Direct endpoint refresh support.

The measured server source is 182cb234. Build it with its pinned Rust toolchain:

Bash
git clone https://github.com/Ambiguous-Interactive/signal-fish-server.git
cd signal-fish-server
git checkout 182cb234c7aa601be3c2aad881948b3c5b853aaa
cargo build --locked --bin signal-fish-server
cp config.example.json config.json
python3 - <<'PY'
import json
from pathlib import Path
path = Path("config.json")
config = json.loads(path.read_text())
config["session"]["default_topology"] = "host"
config["session"]["enable_webrtc"] = False
config["protocol"]["enable_rkyv_game_data"] = True
config["protocol"]["enable_protobuf_game_data"] = True
config["rate_limit"]["max_room_creations"] = 1000
config["rate_limit"]["max_join_attempts"] = 1000
path.write_text(json.dumps(config))
PY
./target/debug/signal-fish-server

From this client repository, in a second terminal:

Bash
godot --headless --path . --script tests/smoke/run_live_server_acceptance.gd

An optional argument after -- selects the server base URL, for example -- ws://127.0.0.1:3536. Success prints LIVE_SERVER_ACCEPTANCE_PASS and exits zero. Each wait has an eight-second deadline and the drill has a two-minute watchdog. A failed phase exits nonzero.

Godot 4.3 on Linux arm64 passed these live checks:

  • Spectator baseline state is installed before its signal. Live player join/leave, lobby readiness, GameStarting and gameplay preserve spectator role. Voluntary leave clears the room and returns to authenticated state.
  • Gameplay reaches spectators with the player's sender ID: v2/v3 JSON, MessagePack raw bytes and decoded values, and opt-in v3 rkyv/protobuf bytes. Raw binary signals carry the negotiated encoding.
  • Two Direct clients receive each fresh shared plan after a host endpoint change, endpoint loss with a successor, and loss of the final endpoint. The Relay reset clears host, endpoint, peers and ICE state. Relay gameplay still arrives. Duplicate and non-host metadata keep the current generation; Pong replies provide processing barriers on both connections.

Additional drills use the same server setup:

Bash
godot --headless --path . --script tests/smoke/run_room_transition_acceptance.gd
godot --headless --path . --script tests/smoke/run_live_transport_reports.gd

The room transition drill queues 80 real receive packets before dispatch, then pipelines spectator leave and join on one connection. It drains one packet per poll and checks terminal/baseline order, payload order and sender, room identity, new roster, and fresh gameplay after rejoining the first room. It covers v2/v3 JSON and raw MessagePack. It also overlaps Direct host endpoint updates with another player's join and leave. Current players converge on one fresh endpoint and generation; each observed plan uses the current roster. These overlapping socket requests do not force internal server race barriers.

The transport report drill uses a v3 reporter, a relay-only v3 observer, and a v2 observer. It checks both state-transition orders, duplicate suppression, self/v2 exclusion, and fresh report state after reconnect. Gameplay on the same connection provides a processing barrier and confirms relay delivery. It covers serial WebSocket dispatch. The controlled lifecycle drill below covers concurrent embedded-host dispatch. On the older pinned server, this drill exposed the missing peer metadata. The roster recovery drill qualifies the repair.

All three drills passed on Godot 4.3 Linux arm64 against the pinned server above. They accept the same optional base URL. Each phase has an eight-second deadline; the room transition watchdog is two minutes and the transport report watchdog is one minute. A failed phase exits nonzero.

The recovery drill adds spectator authority, readiness, detach replay, and Direct reconnect histories:

Bash
godot --headless --path . --script tests/smoke/run_live_recovery_acceptance.gd

Against the same pinned server, Godot 4.3 Linux arm64 passed four spectator cases: v2/v3 observers with JSON/raw MessagePack and a v3 reconnecting player. Observers receive authority release/transfer, current cached readiness and peer return notifications while retaining the spectator role. After a second player disconnect, a viewer leaves. The viewer receives one terminal reply with room identity and reason, clears its session and retained identity, and connected players remove it from their roster. The returning player receives one decoded detach event in replay with the same identity, reason and terminal roster. Replay emits no live detach signal. A fresh spectator join receives the restored roster and gameplay from the returning player. The pinned server pre-issues reconnect credentials only on v3; these cases do not claim v2 directed reconnect coverage.

Two further reconnect cycles overlap a Direct host endpoint update with peer reconnect. Both clients converge on the new endpoint and shared generation. Each fresh plan follows the reconnect baseline, the current credential rotates, and relay gameplay resumes. These are overlapping socket requests; they do not force an internal publication race. The roster recovery drill checks incumbent metadata on the newer server.

The recovery drill uses the same optional base URL, eight-second waits and two-minute watchdog. Success prints LIVE_SERVER_ACCEPTANCE_PASS; any failed phase exits nonzero. It passed against the pinned server above. Ordinary voluntary detach and replay are covered. The publication drill below covers correlated recovery results and authoritative removal for #356. The fault and drain drill below covers both local completion panic points and terminal-before-close ordering.

The controlled publication and server delivery-pressure drills below cover concurrent publication and queue policies. These drills are opt-in and outside all. The mesh drill checks native and browser WebRTC packet boundaries separately.

The pressure drill adds two-sender fan-in and selected-host departure:

Bash
godot --headless --path . --script tests/smoke/run_live_pressure_acceptance.gd

It passed on Godot 4.3 Linux arm64 against server 182cb234. Four v2/v3 JSON/raw MessagePack cases queue 80 real socket frames at an incumbent spectator while a second spectator joins. Dispatch drains one frame per poll. Checks cover exact sender counts, per-sender order, one membership delta, baseline-before-data, cached rosters and fresh gameplay. v2 carries the full roster in membership deltas; v3 carries compact deltas with an empty roster array (src/websocket/sending.rs at the pinned server commit). This measures client receive backlog. The server delivery-pressure drill below covers queue saturation and latest/volatile delivery.

A three-player Direct case removes the selected host. Both survivors receive a fresh shared generation and the successor's endpoint. Removing the remaining endpoints produces a shared Relay reset with no host, endpoint, peers or ICE. Plan callbacks see the reset and relay gameplay succeeds. Controlled publication is covered by the drill below. The publication drill below checks game-owned Direct socket replacement and teardown.

For #356, apply the local-only spectator_detach_fault.patch to an isolated checkout of server 182cb234c7aa601be3c2aad881948b3c5b853aaa. Build with Rust 1.94.0. The patch inserts assertions inside the existing caught local-completion transaction. Use it only for acceptance. It adds no fault switch to the shipped addon.

Bash
# From the isolated server checkout, with the config above:
git apply /path/to/signal-fish-client-godot/tests/fixtures/spectator_detach_fault.patch
cargo build --locked --bin signal-fish-server
SF_GODOT_DETACH_FAULT=after_remove SIGNAL_FISH__SERVER__DRAIN_GRACE_SECS=2 \
  target/debug/signal-fish-server
# In another terminal, from the Godot client checkout:
godot --headless --path . --script tests/smoke/run_live_recovery_acceptance.gd

Repeat with SF_GODOT_DETACH_FAULT=after_local. Each mode passed all four spectator recovery cases and the two Direct reconnect cycles. Server logs confirmed the chosen fault and local recovery. The existing recovery oracle checks exactly one terminal reply, live roster removal and replay detach, room identity, reason, retained identity cleanup and fresh spectator gameplay. These injections exercise the upstream repair; a request-triggered panic is unproven. The controlled drill below covers registration retry/exhaustion. The publication drill below covers authoritative removal.

The same isolated server can then run the drain case:

Bash
godot --headless --path . --script tests/smoke/run_live_pressure_acceptance.gd \
  -- ws://127.0.0.1:3536 drain

When it prints LIVE_DRAIN_READY, send SIGTERM to that server process within eight seconds. The drill has already queued the spectator terminal reply at the engine peer. Both fault modes passed the exact signal order SpectatorLeft, GoingAway, disconnected with close code 4000 and cleared room/identity state. The server uses a two-second grace for this drill. The publication drill below covers recovered correlated payload decoding; SDK room-operation admission remains unsupported. These drills are opt-in.

Active gameplay shutdown

Use this POSIX-only local control for #482, #403 and #355. It starts and stops its own server processes. It runs native and Web seats with 4 KiB relay bodies and exact mesh packets with 4 KiB bodies in all three transfer modes.

Build server commit ee20b51ee605b09cdb448615f7500f6877d8c6e1 or a later source containing the shutdown close repair. Keep the build log and source state with the executable hash. Native runs need Godot and the native WebRTC extension. Web runs need matching Godot Web export templates and Playwright Chromium. The default all runtime needs both sets. Use a local server config with WebRTC enabled and empty ICE servers, as in docs/mesh-guide.md.

Bash
python scripts/check-active-shutdown.py \
  --server-binary /path/to/signal-fish-server \
  --server-config /path/to/local/config.json \
  --server-revision ee20b51ee605b09cdb448615f7500f6877d8c6e1 \
  --native-extension /path/to/webrtc_native \
  --godot /path/to/godot --chromium /path/to/chromium \
  --output /tmp/active-gameplay-shutdown

The output directory must be new. --runtime native|web and --mode graceful|reset|stall select a control. The driver copies the config, selects a local port and sets a three-second server drain. Other config values remain supplied by the caller. The declared server revision is recorded beside the executable hash; the driver cannot prove how an executable was built. The default relay rate is four packets per frame per seat. --relay-packets-per-frame 1 selects a slower comparison. Keep its result separate from the original offered rate.

For a native engine built for another CPU, add --runner /path/to/emulator and repeat --runner-arg=VALUE for each argument. For QEMU x86-64 on ARM64, use --runner /usr/bin/qemu-x86_64 --runner-arg=-L --runner-arg=/usr/x86_64-linux-gnu. This requires --runtime native and --import-godot /path/to/native-editor. Only gameplay runs through the emulator. The importer runs directly. Use an extension bundle containing libraries for both CPUs. Provenance records the emulator hash and arguments beside the engine, extension and importer hashes. Keep emulation results separate from physical hardware results.

Each control first exchanges 32 exact mesh packets per mode and direction. The RPC peer caches an unreliable packet before a later reliable arrival. The drill checks both packets' bytes and modes, then caches another packet for retirement. Relay sends stay active through graceful drain.

  • graceful: SIGTERM must yield GoingAway, then one disconnected(4000, "server_shutdown") per client.
  • reset: SIGKILL must yield an abnormal close or a send-side transport failure. Native and Web raw close values can differ.
  • stall: SIGSTOP holds the server while each client calls close(). The two-second close watchdog must emit heartbeat close timeout.

Every control checks that the mesh is empty, the cached receiver cannot return or send data, and terminal signals stay single. Graceful and reset controls check automatic retry eligibility, then disable retries for a controlled fresh login. User close must leave retries stopped. After server restart, a new plan and receiver must exchange every mesh packet again. The old receiver stays empty.

Check every send result. A Web send can return ERR_UNCONFIGURED while a close frame is pending. The drill records that refusal and stops application sends for that seat. Graceful shutdown still requires the exact semantic close. Relay receives require exact bytes, sender and sequence for the delivered prefix. Shutdown may discard the remaining relay tail.

Each case retains godot.log, server/server.log, assessment.json, import logs and effective config. Web cases also retain export logs. Root provenance records source and executable hashes. Failed attempts remain in their original output directories. These are local acceptance controls; the remote Web lane covers official Web 4.3, 4.4.1 and 4.7.2, and the remote native lane covers official 4.4.1 and 4.7.2 plus the pinned repaired 4.3 backport. Other engine versions and extension builds need separate qualification.

On Linux ARM64, one six-case cohort passes with official Godot 4.7.2, Chromium 156.0.8078.4 and server ee20b51e. Native seats use the opt-in repaired extension described in docs/mesh-guide.md. The server executable SHA-256 is 4f9c94a766864be0a151263b43230518e62910c4dfe9b7b8d4defbcde82fbdd3. The local reports are under /tmp/session452-final-matrix. Later cold-editor imports abort before gameplay (#447). Those driver attempts remain failed. Direct runtime boots of the exact prepared projects pass the native graceful and nonresponding controls. The import failure remains open.

A Godot 4.3 native control with the earlier extension preserves shutdown metadata and fresh mesh gameplay, but fails on a WebSocket receive-buffer overflow and relay sequence gap (#485). Its reset and nonresponding controls pass. The original failure remains under /tmp/session452-native43-original. A separate one-packet-per-frame control passes every check. The slower pass leaves the original reliable receive failure open. These results establish the semantic closes on those local paths. A delayed browser control below exposes a separate send/close race. Other engine coverage and extension builds remain open in #482, #403 and #355.

Remote-path qualification (Web)

The Gameplay Shutdown Remote lane runs the three Web controls through a credential-free Cloudflare quick tunnel. The server keeps binding locally; both seats connect over wss:// through the Cloudflare edge, so every relay message, the GoingAway advisory and the semantic close cross a real remote path. The driver takes --service-port plus --service-public-origin for this; the origin cannot be combined with the delayed local proxy. Four consecutive green runs of both jobs - the post-merge main run 41905622779 plus three branch dispatches of the same lane (41898423659, 41900140061, 41904306119) - pinned the remote shutdown signature, so the lane carries a weekly scheduled run (Mondays 13:00 UTC, staggered after the WebSocket Remote Path watch) with the same pinned inputs. The weekly Web run covers official Godot 4.7.2, 4.4.1 and 4.3; the native run covers 4.7.2 and 4.4.1. Each job takes its own dispatch list in web_engine_version and native_engine_version (space or comma separated) to qualify another target or re-run one engine. Dispatch runs keep on-demand qualification available; GitHub disables schedule triggers after ~60 days without repo activity. A red run triages from the run page: a drill failure reads the run summary alone (the known #447 import-abort attribution first; the native loop already re-dispatches import-unstable attempts and retains each), while a tunnel-provisioning failure points at the failed tunnel step (run 41897404439 is the transient record).

Official 4.4.1 Web qualified in the two-engine dispatch 41984709949 (Chromium 156.0.8078.4, 32 KiB application-ACK window). All three controls passed: graceful delivered GoingAway then exactly one 4000/server_shutdown close per seat, reset closed 1006, and stall ended with the heartbeat close timeout. Both gameplay generations completed in every mode, each case counted two service starts and one shutdown request, and provenance recorded the tunnel origin. Official 4.7.2 repeated green in the same run.

Official 4.3 Web qualified in dispatch 41991293515 (Chromium 156.0.8078.4, 32 KiB application-ACK window). All three controls passed: graceful delivered GoingAway then exactly one 4000/server_shutdown close per seat, reset closed 1006, and stall ended with the heartbeat close timeout. Both gameplay generations completed in every mode, and each case counted two service starts and one shutdown request. The 4.7.2 and 4.4.1 Web jobs repeated green in the same run. The native job keeps its 4.7.2 and 4.4.1 pair: official 4.3 native still hits the #485 receive loss at the original uncredited rate.

Remote-path qualification (native)

The lane's native job runs the three native controls through the same kind of quick tunnel. One matrix entry builds the pinned repaired 4.3 backport from source (below); the rest are official releases. The official editor imports each case project and runs it headless; the pinned official WebRTC GDExtension backs its peer connections. The lane pins release 1.2.2-stable by zip SHA-256; that tag is commit 62135fc034cbdc5aee177136e526e3ead3cc563a, the source revision other qualifications build from. The official release binary itself is new qualification ground: earlier native #482 cohorts ran opt-in repaired extension builds. The gate compares each case's staged extension library hash against the release hash after the drill, so a substituted library is detected and fails the qualification; the exact gameplay checks are what qualify the peer behavior. The native seats use the original uncredited workload (four 4096-byte relay bodies per frame), the control the local native qualifications used; a relay loss over the edge is new #488-family data, not a close-metadata failure. The per-case cold-editor import remains the known-open #447 abort hazard; the gate attributes an import abort so a red run is readable as engine-import data before it is read as remote-path data. The lane also re-dispatches import-unstable attempts (retaining each) and gives every editor invocation a fresh HOME with --fresh-home: the captured aborts all land right after the editor loads its layout, which reads shared state under the caller's HOME.

Official 4.4.1 native qualified in the same dispatch (41984709949) with the pinned release extension (library SHA-256 d99224f5411c5083f1d0932f2ed30b0b68f5f697be025671ff0083015f06a033). All three controls passed at the original uncredited rate: graceful delivered GoingAway then exactly one 4000/server_shutdown close per seat, reset reported the raw abnormal close -1, and stall ended with the heartbeat close timeout. Both gameplay generations completed in every mode, and each case counted two service starts and one shutdown request.

The native list also carries the pinned repaired 4.3 backport (77dcf97d82cbfe4e4615475fa52ca03da645dbd8, scripts/backport-websocket-43.py). The job builds a template_debug x86-64 engine with GDScript, WebSocket, WebRTC and mbedtls, imports each case with the official 4.3 editor, and runs the same three controls at the original uncredited rate (four 4096-byte bodies per frame, no credit). This is the 4.3 engine that holds the #485 receive boundary; official 4.3 does not. Dispatch 42076926873 passed all three: graceful delivered GoingAway then exactly one 4000/server_shutdown close per seat, reset reported -1, and stall ended with the heartbeat timeout. Both generations completed in every mode, and each case counted two starts and one shutdown request. The gate compares the run's engine hash with the freshly built binary and checks the custom-build version string and the extension library; this run used SHA-256 8aebdbce741747e08ba00ec694764e38d9d311ee94e5193d582c447197d831f3. The first dispatch (42074629297) passed every control but failed its gate on the 4.3 version string (4.3-stable (custom_build)); that attempt stays retained. A dispatch sets native_repaired_4_3 to skip the build. Upstream adoption stays open in #536.

Native WebSocket receive boundary

Issue #485 is reproducible with WebSocketPeer and the local smoke server. This native control needs no Signal Fish server or WebRTC extension:

Bash
godot --headless --path . \
  --script tests/smoke/run_websocket_receive_boundary.gd

It writes 64 exact 4096-byte messages, holds client polling for 100 ms, then drains. A later sentinel exposes an interior gap after a dropped burst tail. Every write must succeed. Every received packet must match its bytes, string mode and sequence. The final count must be 65. Engine errors remain failures. The watchdog covers the five-second connection window, each batch hold/read window and the final sentinel read. It is opt-in because the original Godot 4.3 failure must remain visible.

Use -- --credit=32768 to split the same finite work into batches bounded by local consumer body bytes. Use -- --capacity=524288 for a larger byte buffer, or add --max-queued-packets=2 to isolate the packet-count boundary. This native byte credit bounds bytes only; it does not bound packet count. Credit uses direct local consumer observations; it adds no network ACK protocol. The script prints each poll, packet and final verdict. Keep logs, commands, source hashes and engine hashes for every attempt.

On Linux ARM64, the new controls retain these results:

Engine Default buffer 512 KiB buffer 512 KiB, two packets 32 KiB body credit
4.3 Fails: 16/65 Pass: 65/65 Fails: 3/65 Pass: 65/65
4.4.1 Pass: 65/65 Pass: 65/65 Pass: 65/65 Pass: 65/65
4.7.2 Pass: 65/65 Pass: 65/65 Pass: 65/65 Pass: 65/65
4.3 backport Pass: 65/65 Pass: 65/65 Pass: 65/65 Pass: 65/65

The default byte setting is 65535 in all three engines. Matching the newer engines to 2048 queued packets also passes. The changed default packet count alone does not explain the repair. Godot's unbuffered receive repair limits socket reads to remaining queue space. Godot 4.4.1 includes this repair and later fragmented-message fixes. An SDK drain cannot recover bytes already dropped inside a Godot 4.3 engine poll. A larger finite buffer only moves that boundary. The retained 4.3 failure still needs an engine upgrade or qualified backport for a receive-boundary repair.

Add -- --fragment-bytes=127 to split each message into WebSocket TEXT and CONTINUATION frames. Zero preserves single-frame messages. Message bytes, credit, capacity and the later sentinel stay unchanged. TCP may coalesce the writes. Godot 4.4-beta1 passes the original control but truncates fragmented messages: a 512 KiB run receives 65/65 messages with 127-byte bodies instead of 4096 bytes. The exact-byte check fails from packet zero. Godot 4.4.1 and 4.7.2 pass the same fragmented controls. Add --message-type=binary for bodies containing all byte values, or --message-type=alternating to alternate text and binary messages. Both bytes and packet type must match the offered order. The default text workload stays unchanged. A 4.3 backport also needs the message-completion callback repair, the fragmented-message assembly repair, and the closing-buffer lifetime repair. The assembly change alone still commits individual fragments. Clearing the receive buffers before the close handshake blocks its socket reads. The beta is a historical negative control; it adds no support promise.

Add --fragment-control=ping or --fragment-control=pong with a positive fragment size to place a nonempty control frame inside each fragmented message (#491). Control bytes must stay out of the delivered text. Official Godot 4.4.1 and 4.7.2 fail this exact-byte control even when all 65 packets arrive. The buffered Godot 4.3 capacity control preserves the message bytes.

Apply the experimental native backport to a clean Godot 4.3-stable source checkout with:

Bash
python scripts/backport-websocket-43.py --source /path/to/godot \
  --report /tmp/websocket-backport.json

The script pins the source revision and all four upstream commits. It records patch and source hashes, rejects conflicting edits, and supports --check. Build the engine separately and retain its command, log and binary hash. The experimental patch also keeps interleaved Ping/Pong bytes out of the fragmented message and clears pending assembly on connection replacement. This patch does not change the Web callback receive boundary. Issue #492 covers a separate direct-peer reuse failure. Run godot --headless --path . --script tests/smoke/run_websocket_fragment_reuse.gd to replace a closing peer after an unfinished fragment. The fresh text must be exactly five bytes. Official 4.4.¼.7.2 retain stale assembly length and return an empty packet with a buffer error. The SDK creates a fresh engine peer for each session; this direct-reuse result does not establish an SDK reconnect defect. Pass script options after Godot's -- separator:

Bash
godot --headless --path . \
  --script tests/smoke/run_websocket_fragment_reuse.gd -- \
  --partial-type=binary --fresh-type=binary

Choose the partial and fresh message types independently. The fresh binary body includes zero and non-UTF-8 bytes.

Separate experimental patches target Godot 4.4.1-stable 49a5bc7b616bd04689a2c89e89bda41f50241464, 4.5.2-stable 6ce3de25aa58466e14ef354703ba8d9791a417da, 4.6.3-stable 35e80b3a8822a9df9be390814b62f44c0a9c69e8, 4.7.2-stable ed1daf0bf001b61586d9930840f2f1394092c079 and master 95c0fa5fe8a019adc29d074c16b64469b04571f7:

Bash
python scripts/repair-websocket-stable.py --source /path/to/godot-stable \
  --report /tmp/websocket-assembly.json

Each patch repairs receive progress, frame assembly and replacement cleanup. The applicator checks the exact revision, selects its patch, rejects conflicting changes and records source and patch hashes. --check leaves source unchanged. Build and qualify the patched engine before use. The master pin carries the same repair submitted upstream (godotengine/godot#125066); its four defects and evidence map to #491, #492, #496 and #498. Official 4.4.1 and 4.7.2 still fail the mixed binary interleaved controls and every partial/fresh text/binary reuse pair. Further engine builds and remote gameplay remain open.

A weekly scheduled run (Mondays 09:00 UTC, WebSocket Master Pin) keeps the master pin aligned with upstream. It compares the pin with current godotengine/godot master and prints the state of godotengine/godot#125066 for review tracking. When master moves, it mirrors the patched files at the new revision and checks that the bundled patch still applies. A moved master with an applying patch passes. A red run names the outcome: the receive path drifted (rebase the patch, refresh the submission) or the repair was adopted in full (the patch reverse-applies; requalify the adopted revision, then retire the pin).

The experimental 4.5.2 Linux ARM64 template passes all seven standalone reproductions below. Official 4.5.2 fails the four defect cases and passes the three comparison controls. These checks cover fragmented Ping/Pong bytes, CLOSING peer reuse, full-buffer completion and split Unicode close reasons (#491, #492, #496, #498). The build also passes all 181 paced and large-boundary cases, normal WebSocket smoke and all three delayed active shutdown/recovery modes. Other build targets and remote networks remain open.

The experimental 4.6.3 Linux ARM64 template also passes all seven standalone commands. A matched unpatched build from the same source revision and build options fails all four defect commands and passes the three comparison controls. The checksum-verified 4.6.3 release asset fails 108 of the 181 receive cases. That asset reports commit 7d41c59c4; the source builds use 35e80b3a8. Keep these baselines separate. The repaired build passes all 181 receive cases, normal WebSocket smoke and all three isolated active shutdown/recovery modes. An initial concurrent graceful attempt hit outbound send backpressure; the initial stall setup aborted during editor import (exit -6; #447). Retain both failed attempts. The isolated passes do not explain either failure.

The same 4.6.3 repair also passes all 181 receive cases, all seven standalone commands and SDK WebSocket smoke on Linux x86-64 under QEMU 8.2.2 on ARM64. The matched unpatched x86-64 build uses the same source revision and compiler options. It reproduces packet corruption, stale reuse state, a full-buffer stall and code 1007 for a valid Unicode close. All three comparison controls pass. These are local emulation results. Physical x86-64 hardware and remote networks remain open.

The 4.7.2 repair also passes all seven standalone commands on Linux x86-64 under QEMU 8.2.2 on ARM64. Matched builds use revision ed1daf0bf001b61586d9930840f2f1394092c079 and identical compiler options. The unpatched build reproduces all four defects and passes all three comparison controls. This covers fragmented Ping/Pong bytes (#491), peer reuse (#492), full-buffer completion (#496) and Unicode close reasons (#498). The repair passes all 181 receive cases, SDK WebSocket smoke and all three active shutdown/recovery modes (#482). The active drill uses 50 ms proxy delay, 127-byte chunks and a 100 ms receive pause with the original deadlines. Each mode preserves exact mesh traffic across both connection generations and an exact delivered relay prefix. Physical x86-64 hardware, remote networks and upstream adoption remain open.

For 4.6.3 and 4.7.2 template builds, add disable_path_overrides=no to the SCons command to enable --path and --script for these fixtures. Use the matching official editor to import the test project before running a template build. The active shutdown drill also needs module_multiplayer_enabled=yes alongside GDScript, WebSocket and WebRTC.

To check the four bundled stable patches against local pinned sources, run:

Bash
PYTHONPATH=tests \
SF_WEBSOCKET_GODOT_44_SOURCE=/path/to/godot-4.4.1 \
SF_WEBSOCKET_GODOT_45_SOURCE=/path/to/godot-4.5.2 \
SF_WEBSOCKET_GODOT_46_SOURCE=/path/to/godot-4.6.3 \
SF_WEBSOCKET_GODOT_47_SOURCE=/path/to/godot-4.7.2 \
python -m unittest test_websocket_browser_observation.BackportApplicatorTests.test_bundled_patches_apply_to_pinned_sources

This opt-in test uses temporary local clones to check, apply and repeat each real patch. The source checkouts stay unchanged. Normal CI skips this test.

The locally built 4.4.1 native template passes 31 controls with no engine errors: text, binary and alternating messages across no-control/Ping/Pong, default byte limits, two queued packets and local 32 KiB credit, plus all four partial/fresh message-type pairs. Every burst delivers 65 exact packets with matching types. Reuse delivers exactly five fresh bytes. The same build passes the standard WebSocket smoke with client/server close handshakes. Its active SDK drill passes semantic shutdown, reset and nonresponding controls with the original uncredited four-per-frame sends, 50 ms TCP delay, 127-byte proxy writes and a 100 ms polling hold. Exact relay prefixes, cached mesh retirement and fresh gameplay receipts remain required. These results qualify the pinned Linux ARM64 build and local workload.

The locally built 4.7.2 native template passes the same 31 controls, close smoke and all three active SDK shutdown modes with the same uncredited sends, TCP delay, proxy writes and polling hold. Every burst delivers 65 exact 4096-byte packets; each reuse pair delivers five fresh bytes with the correct type. A matched official 4.7.2 run passes nine no-control burst cases and fails all 18 interleaved-control cases and four reuse pairs. Its active shutdown controls pass. These results qualify the tested Linux ARM64 build and local workload. Remote paths, other targets and upstream adoption remain open.

Use --import-godot /path/to/official-godot-4.3 in the active-shutdown driver to import with an editor and run a native template build via --godot. Both engine binaries are hashed in the retained results.

The locally built 4.3 template passes all three active-shutdown modes with the original uncredited four-per-frame sends, 50 ms TCP delay, 127-byte proxy writes and a 100 ms polling hold. Receive capacity stays at 65535 bytes. Exact relay prefixes, semantic/reset/nonresponding close outcomes, cached mesh retirement and fresh exact gameplay remain required. This qualifies the tested native build and local workload; upstream and remote qualification remain open. A new matched official 4.3 shutdown control also passes; that passing comparison does not explain the retained overflow history. The finite burst controls keep the native receive-boundary failure explicit.

Full fragmented-message receive limit

A fragmented message can fill the native receive buffer before its final empty continuation arrives (#496). Official Godot 4.4.¼.7.2 and the earlier experimental native patches stall with an open connection and no complete packet. Run:

Bash
godot --headless --path . \
  --script tests/smoke/run_websocket_fragment_limit.gd

The fixture fills the 4095-byte buffer with an unfinished binary message, then sends an empty final continuation and a separate two-byte binary message. Both messages must arrive with exact bytes and types. Use -- --message-type=text --fragment-control=ping for text and an exact Pong response. --fragment-control=pong checks incoming Pong separation. --capacity=65535 checks the default byte limit. --oversize=true adds one continuation byte: the peer must close with code 1009 and return no packet. Engine buffer errors fail every control.

The updated experimental patches permit one byte per socket read when an unfinished message fills the payload buffer and no complete packet is queued. This lets empty continuations and controls progress without reading into the next message. The bundled wslay change tracks cumulative message length in unbuffered mode, so an extra continuation byte still exceeds the configured limit. The 4.3 patch also adopts the pinned 4.4.1 close handling for a stopped parser, so a rejected message reaches a terminal close state. Apply the current patch to a clean pinned source checkout and rebuild. Earlier patched binaries do not contain this repair.

The burst fixture also accepts --body-bytes=4095 --capacity=4095 and --empty-fragments=true --fragment-bytes=127. Empty fragments surround each message; the default 4096-byte workload stays available. The reuse fixture accepts --replacement=closing|aborted|handshake and --queued-before-partial=true. It checks that a complete unread packet and unfinished fragment are retired before the fresh connection's five-byte message.

The rebuilt Linux ARM64 templates for 4.3, 4.4.1 and 4.7.2 each pass 84 receive/reuse controls, WebSocket close smoke and all three active SDK shutdown modes. These local results extend #485, #491 and #492. Remote networks and upstream adoption remain open.

Paced native WebSocket reads and close reasons

Use the opt-in native runner to split frame headers and bodies across local TCP writes. It covers full-buffer completion (#496), Ping/Pong separation (#491), peer replacement (#492), and valid Unicode close reasons (#498):

Bash
python scripts/check-websocket-native-receive.py \
  --godot /path/to/repaired-template --importer /path/to/godot-editor \
  --output /tmp/native-receive-attempt

Use a new output directory for every attempt. Reports retain commands, logs, fixture hashes, executable hashes, exact result records and timeout failures. On native Linux, add a syscall read control to keep TCP coalescing from joining all the paced writes into one receive:

Bash
python scripts/check-websocket-native-receive.py \
  --godot /path/to/repaired-template --importer /path/to/godot-editor \
  --recv-cap-bytes=3 --output /tmp/native-receive-cap3

This option requires cc and a dynamically linked Linux engine. The runner builds a local LD_PRELOAD helper that limits each libc recv call on both fixture endpoints. The editor import runs without the helper. Choose an output path without whitespace or colons. The option supports native execution; QEMU runs use the write controls above.

Each attempt keeps the helper source, compiled library, build command, hashes, child environment and per-case read counters. A case requires observed limited reads, a maximum read size within the chosen cap and all existing packet/close checks. Missing observations fail the attempt. Existing workloads and deadlines stay unchanged. These controls qualify local read boundaries; remote paths need separate runs.

Use --case=read-control-none, --case=read-control-ping and --case=read-control-pong for the roomy-buffer comparison. These three cases keep each fragmented 4095-byte message below the receive capacity. They isolate control-frame assembly from the full-buffer completion cases.

On Linux ARM64, official 4.7.2 fails both roomy Ping/Pong controls with a three-byte read cap: a 4095-byte message arrives with 4114 bytes. The no-control case passes. With the same cap, a complete-frame Unicode close write becomes code 1007 in both directions; ASCII and empty reasons pass. The repaired 4.3, 4.4.1 and 4.7.2 builds pass the roomy controls. These results qualify local syscall boundaries.

To test a cross-compiled engine with QEMU, use an explicit runner:

Bash
python scripts/check-websocket-native-receive.py \
  --godot /path/to/x86-64-template --importer /path/to/native-editor \
  --runner /usr/bin/qemu-x86_64 \
  --runner-arg=-L --runner-arg=/usr/x86_64-linux-gnu \
  --output /tmp/native-receive-x86-attempt

The importer runs on the host. Reports keep the target engine hash, runner hash and runner arguments separately. Keep the cross compiler and sysroot versions with the build record. Emulation results apply to local emulation; physical hardware and remote networks need separate qualification.

Use --suite=assembly, --suite=close or --suite=boundaries to select a group; --case=NAME selects individual cases. The full matrix has 184 cases. The boundaries group adds 55 cases for large gameplay and control bodies:

  • 131071-byte messages use 64-bit frame lengths. Text, binary and Unicode bodies require exact full-buffer completion or code 1009 for an extra byte.
  • Unicode characters span continuation boundaries with up to three tail bytes. Interleaved Ping/Pong bodies contain all byte values from 0 through 124. Ping requires an exact 125-byte Pong.
  • Maximum-length close reasons contain 123 UTF-8 bytes. ASCII and mixed two-/three-/four-byte text are checked in both directions with an empty packet queue or an unfinished full-buffer message. Client-initiated close frames must preserve the exact reason bytes as well as the code.

The receive-limit fixture accepts --capacity=131071, --message-type=unicode, --tail-bytes=1|2|3 and --control-bytes=125. The close fixture accepts --reason=max-ascii|max-mixed and --capacity=127|131071. These optional controls retain the original cases.

The three receive/reuse fixtures accept --wire-chunk-bytes=1 and --wire-delay-ms=10. The sender queues frame bytes, writes at most one chunk per server poll, honors partial writes and retires queued output when the connection changes. Delay alone selects 65536-byte chunks. The default sender writes each complete frame immediately. Reports include successful writes, bytes, split headers, pending bytes, retired bytes and write errors. TCP can coalesce writes; this is local scheduling qualification.

The new close fixture checks exact code 3400 and valid empty, ASCII, two-, three-, four-byte and mixed UTF-8 reasons. For a focused reproduction:

Bash
godot --headless --path . \
  --script tests/smoke/run_websocket_close_reason.gd -- \
  --wire-chunk-bytes=3 --wire-delay-ms=10 --reason=two --origin=server

Use --origin=client for a client close with a paced server echo and --full-fragment=true to close with an unfinished 31-byte message filling the receive buffer. Both directions require the exact close reason, terminal state, no application packet and the matching client close code.

Bundled wslay skips the first two close-payload bytes during every receive chunk's UTF-8 validation. Those status bytes occur once per close frame. A valid Unicode reason split across reads can therefore produce code 1007. The current experimental 4.¾.4.¼.7.2 patches skip only the status bytes that remain at the accumulated payload offset. Rebuild from the current patches; earlier repaired binaries still contain this close-reason defect. Upstream adoption, other targets and remote networks remain open.

Standalone upstream reproductions

Create a small project for engine bug reports:

Bash
python scripts/check-websocket-native-receive.py \
  --export-repro /tmp/websocket-repro.zip

Extract the ZIP and follow README.txt. The project needs only Godot and uses a local raw TCP server. It includes exact-byte Ping/Pong assembly, peer replacement, full-buffer completion and split Unicode close controls, plus passing comparison commands. SHA256SUMS pins every input. The exporter refuses an existing archive and accepts no runtime options.

Godot closed the upstream reports because it does not accept AI-authored issues. This repository keeps the maintained record in .llm/research/godot-websocket-receive-defects.md:

Failure Local record
Ping/Pong bytes enter fragmented messages (#491) defect 1
Closing peer replacement retains old state (#492) defect 2
Full-buffer empty continuation stalls (#496) defect 3
Split Unicode close reasons fail (#498) defect 4

The extracted project's seven commands were checked on official and repaired 4.¾.4.¼.7.2 Linux ARM64 engines. All three repaired builds pass every command. Official 4.4.¼.7.2 fail the four defect cases and pass the three comparison controls. Buffered 4.3 passes assembly/reuse; its full-buffer case drops the following packet. Current Godot master 7be59b46 retains the four source paths; no master runtime result is claimed. Upstream adoption, other targets and remote networks remain open.

The driver can also run those seven commands itself and retain one report:

Bash
python scripts/check-websocket-native-receive.py \
  --godot /path/to/engine --importer /path/to/editor \
  --run-repro --output /tmp/standalone-run

standalone-report.json pins engine and importer hashes and records optional emulator paths and arguments. It records each command's exit and result records. Each command's log sits beside the report as <name>.log. An engine that crashes a command leaves no result, or an empty record when its result line is cut short. Either way the gap stays visible. The driver exits nonzero when the fixture import fails, including an import that exits 0 after logging an ERROR: line. It writes the report before it exits. Only the import sets the exit status: a failing reproduction command still exits 0, because official engines fail the defect commands by design. Callers compare the report with the expected matrix.

x86-64 CI qualification

The opt-in WebSocket Native x64 workflow builds a pinned engine revision on an x86-64 GitHub runner and qualifies it there. Dispatch it with a pinned stable or master revision to build the repaired engine, run the full paced receive suite with a three-byte read cap, and run the seven standalone commands. Set the matching official release label for the fixture import; a master dispatch uses the nearest official release label because no release matches a master SHA. A build_target input selects linuxbsd (default) or windows; Windows builds cross-compile with mingw-w64, run under Wine, and use the uncapped suite. A repair_kind input selects the repair applicator. stable (default) patches the pinned 4.4.1, 4.5.2, 4.6.3, 4.7.2 and master revisions with repair-websocket-stable.py. 4.3-backport patches 4.3-stable 77dcf97d82cbfe4e4615475fa52ca03da645dbd8 with backport-websocket-43.py; dispatch that run with the matching revision and the 4.3-stable label. The 4.3 remote leg pairs 4.3-backport with run_baseline and remote_path on. The 4.3 build drops the disable_path_overrides SCons flag, which 4.3's SConstruct predates. The lane rejects a stable source whose label names a different major.minor family; a master pin keeps the nearest official release label. A matched unpatched baseline job runs the same seven commands, each keeping exactly one observed result record. The defect commands fail on official 4.4.1, 4.5.2, 4.6.3 and 4.7.2 baselines. Official 4.3 passes the control-bytes and reuse repros unpatched, so its baseline fails the full-buffer and close-unicode commands. The three comparison controls must pass. Both jobs upload build logs, patch reports, engine hashes and per-command evidence. A remote_path input adds the remote-network leg: each variant fronts a Cloudflare quick tunnel and drives the ten pinned receive cases over wss:// (see Remote path qualification). The build enables the mbedtls module so the engine can connect over TLS at all.

The first run qualified the pinned 4.7.2-stable repair ed1daf0bf001b61586d9930840f2f1394092c079 on a virtualized x86-64 host that executes the engine natively (run 41251544967). The repaired engine passes all 184 paced receive cases with a three-byte read cap; every case observed capped reads. All seven standalone commands pass (#491, #492, #496, #498). Repaired binary SHA-256: c422f1a4b40b5b5f52cb6c87cc7984d661bb4925d625c7b390d7c161bf65a27d. The matched unpatched build from the same source and flags reproduces all four defect commands with one observed result record each. The three comparison controls pass. Unpatched binary SHA-256: e3ffcc28cfa930b4688367ebbabef4037c7255a4ae93d689599c784eaa4c0c6e. SCons 4.11.1 builds both engines with the options listed in the workflow. These results cover the #485 receive suite, paced writes, large boundaries, close reasons, peer replacement and local syscall read caps on x86-64. Keep them separate from QEMU emulation runs. Bare-metal hardware and upstream adoption remain open, and the macOS leg stays with the weekly WebSocket Native macOS watch. The 4.4.¼.5.2/4.6.3 Linux remote legs and the Windows remote dispatch are recorded under Remote path qualification.

The pinned 4.4.1-stable repair 49a5bc7b616bd04689a2c89e89bda41f50241464 passes the same checks on x86-64 (run 41256780905). All 184 paced receive cases pass with capped reads and all seven standalone commands pass (#491, #492, #496, #498). Repaired binary SHA-256: 3db399d851cd054152024c902107521eeb42155845c9b22f07747caa16fe093f. The unpatched baseline reproduces the four defect commands and passes the three controls. Unpatched binary SHA-256: d01959e694c2c4c16855ab1faa4c8d4b4d15f01440d3a20c1c37bb7bf49d92a7.

The pinned 4.5.2-stable repair 6ce3de25aa58466e14ef354703ba8d9791a417da passes the same checks on x86-64 (run 41256782273). All 184 paced receive cases pass with capped reads and all seven standalone commands pass (#491, #492, #496, #498). Repaired binary SHA-256: 03cbf9c4f87f5bab3ca22231992fc7abcb72ae1ca518d57d2a812cdab0eb86f7. The unpatched baseline reproduces the four defect commands and passes the three controls. Unpatched binary SHA-256: 0af7ced3589f5046f9b19d7d1c0fef5b49d2a98142da82758632a8ea7895d4c9.

The pinned 4.6.3-stable repair 35e80b3a8822a9df9be390814b62f44c0a9c69e8 passes the same checks on x86-64 (run 41256784292). All 184 paced receive cases pass with capped reads and all seven standalone commands pass (#491, #492, #496, #498). Repaired binary SHA-256: 7a1432456df8b78940200bf14f5ffc8c49e65cc12986df6814e4518701a59002. The unpatched baseline reproduces the four defect commands and passes the three controls. Unpatched binary SHA-256: 566db2f263e02fcd266a99535142f156133c151d761f9b905cbbae612eeda466.

The pinned 4.3-stable backport 77dcf97d82cbfe4e4615475fa52ca03da645dbd8 passes the same checks on x86-64 (run 41730274982). All 184 paced receive cases pass with capped reads and all seven standalone commands pass (#491, #492, #496, #498). Repaired binary SHA-256: 8aebdbce741747e08ba00ec694764e38d9d311ee94e5193d582c447197d831f3. The unpatched 4.3 baseline passes the three controls, the control-bytes and reuse commands, and fails the full-buffer and close-unicode commands, matching the collected Linux ARM64 baseline results. Unpatched binary SHA-256: 68c4bce9bca9c3947428069ab1d30ad5c2c2ebd6ee5c7acbef9749f293fc70df.

The windows build target qualifies the repair on Windows x86-64 under Wine. The workflow installs current Wine from the WineHQ repository; Ubuntu 24.04 Wine 9.0 cannot start Godot 4.7.x Windows binaries, official or self-built. The pinned 4.7.2-stable repair ed1daf0bf001b61586d9930840f2f1394092c079 passes all 184 receive cases without read caps (run 41273388162). All seven standalone commands pass. The matched unpatched baseline reproduces the four defect commands and passes the three comparison controls (#491, #492, #496, #498). Repaired binary SHA-256: 918124422ff1fa405042e8df31f91bfc49acb3ebe9a186a341a70f261634b3b0. Unpatched binary SHA-256: 759cb4131ba251ee3e93593179a64f40e26d6acfc6d79661af1f9f02ad3e2874.

The pinned 4.5.2-stable repair 6ce3de25aa58466e14ef354703ba8d9791a417da passes the same uncapped checks on Windows x86-64 under Wine (run 41278234161). All 184 paced receive cases pass and all seven standalone commands pass (#491, #492, #496, #498). The unpatched baseline reproduces the four defect commands and passes the three controls. Repaired binary SHA-256: 4266d1d6ddae49e6967b3ae7073525a79f7f847ac5859346c3aedb42a13b5ef4. Unpatched binary SHA-256: daa3a46b4f1f146a4cd2fa96690bea3ba9e773713ce5600f8fbad3697b9decc4.

The pinned 4.6.3-stable repair 35e80b3a8822a9df9be390814b62f44c0a9c69e8 passes the same uncapped checks on Windows x86-64 under Wine (run 41278235547). All 184 paced receive cases pass and all seven standalone commands pass (#491, #492, #496, #498). The unpatched baseline reproduces the four defect commands and passes the three controls. Repaired binary SHA-256: 5fc7e44f8cdd2dbaef133ba13fe61febbcc3d6d5cb2a1e91a7ca7882d64ea85c. Unpatched binary SHA-256: a00ba934fa2255a486a5a87b95682caba39db1524931a3da8d8c7fb10bd1ccc4.

The pinned 4.4.1-stable repair 49a5bc7b616bd04689a2c89e89bda41f50241464 passes the same uncapped checks on Windows x86-64 under Wine (run 41280052786). All 184 paced receive cases pass and all seven standalone commands pass (#491, #492, #496, #498). Repaired binary SHA-256: 90739b2e631da58757dc5fa723ca858fbe3c372a841269cb18b8992cc36305ca. The unpatched baseline in run 41278232278 reproduces the four defect commands and passes the three controls. Unpatched binary SHA-256: f74caff33936d986323b2bff0620cb4d46144a268849d5a241995679ee89778a. An earlier repaired build of the same revision in that run (SHA-256 93713f1b1bee33d9dd356c92c80cb1af04dfb0b6e7d3d6f7c4522f7270d9f6f8) missed the suite's one-second finite-burst deadline once in chunk-burst-alternating-pong: message 40 needed 1.14 s after message 39 while earlier messages averaged about 0.5 s, the deadline expired, and the harness aborted the remaining bursts and offered only the sentinel. All 41 offered messages arrived with exact bytes and mode. The repeat run delivered all 65 packets; the single deadline miss stays open pending recurrence (#520).

A weekly scheduled run (Mondays 08:00 UTC) keeps that watch active: it builds the repaired 4.4.1-stable engine for Windows x86-64 and runs the paced suite plus the seven standalone commands without the baseline job. Dispatch runs keep every input and both variants. Each burst case report carries a delivery summary with the packet count, poll count, largest inter-delivery gap and largest poll interval, parsed from the fixture log. When a batch deadline expires, the burst result records a pending_wait entry with the first missing sequence (or the delivery count when no gap was observed), the elapsed wait and the deadline margin. The batch deadline is three seconds (#520): the healthy Windows x86-64 runs never needed more than 563 ms between consecutive deliveries, and the recorded one-off Wine pacing stall needed 1.14 s, so the former one-second window left under twice the headroom and let a benign stall fail the watch. A deadline miss still names a stall above the window or a lost message; compare both records with the batch deadline before changing any deadline. The suite log prints a delivery line for every passed burst case, and a failing case in either the suite or the standalone log prints a diagnostics line with its result record, so a red run triages from the CI log alone. The uploaded artifacts keep the complete per-case evidence.

Windows-native CI qualification

The windows build target runs under Wine by default, which cannot show that the repair holds on the Windows socket path itself. Dispatch the workflow with build_target=windows and windows_native_runner=true: the Linux job uploads the repaired engine, and a windows-latest job downloads it, installs the official Windows editor through .devcontainer/install-godot.py, and runs the uncapped paced suite plus the seven standalone commands. The gate requires every case and reproduction.

Dispatch 42132635865 (Windows Server 2025, windows-2025-vs2026 image) built the pinned 4.7.2-stable repair ed1daf0bf001b61586d9930840f2f1394092c079 and ran the same engine binary on both paths:

  • Wine leg: 184/184 paced cases and 7/7 standalone commands passed.
  • Native Windows leg: 184/184 paced cases and 7/7 standalone commands passed; both fixture imports exited 0 with no engine errors.
  • Engine SHA-256 51115c67ff60d889c7f79f3b90c88234a96b0d4bab8f05927ad93516e25fde67; extracted official Windows editor executable SHA-256 ab1824f85bfd8e0e4128182c000c4003a3e042245b2967848d089b2a04b22424.

The gate checks the downloaded engine against the uploaded checksum and the suite provenance, so both paths run the same binary. The native leg takes about six minutes after the build. It stays dispatch-only; the weekly Wine watch remains the recurring Wine check, and native Windows runs on dispatch. This qualifies the Windows socket path for the pinned build, not an MSVC rebuild or another engine family.

macOS CI qualification

The WebSocket Native macOS workflow qualifies the same pinned repairs on Apple Silicon (Mondays 11:00 UTC). It builds the repaired engine on a macos-15 runner with the minimal headless feature set (vulkan=no metal=no opengl3=yes), runs the uncapped 184-case paced receive suite and the seven standalone commands. The official macos.universal editor imports the fixtures; its binary runs from inside the extracted Godot.app bundle (the bare bundle binary will not exec on Apple Silicon). Dispatch runs accept a pinned stable or master revision and can add the matched unpatched baseline, which must reproduce the four defect commands (#491, #492, #496, #498) and pass the three controls.

The pinned 4.4.1-stable repair 49a5bc7b616bd04689a2c89e89bda41f50241464 passes all seven standalone commands on macOS arm64 (repaired binary SHA-256 5bfe14836ec8e33390c85cdd145c456819b7240d683f9f3f166f483420a91308). The matched unpatched baseline reproduces the four defect commands and passes the three controls on the same runner, so the platform gap sits in the engine, not the fixtures. The uncapped paced suite holds a measured signature across qualification runs (41510773288, 41516492113), and the workflow asserts it:

  • Twenty stable failures: every one-byte-paced byte-reuse-* case misses the barrier pong (#492 family), and every one-byte-paced boundary-close-1-max-* case never delivers a close code (#498 family). Short close reasons (empty through four-byte Unicode) pass one byte at a time, so the payload-offset close fix itself works on macOS.
  • At most one sporadic chunk-reuse-* failure per run, always the stale-continuation retirement check.

Any drift in either direction fails the run; a repaired engine that starts passing a documented gap names requalification. Requalify against an adopted upstream revision before retiring a documented gap (the resubmission need is tracked in #536). Bare-metal hosts, remote networks and upstream adoption stay open after macOS CI qualification.

Web WebSocket receive boundary

Issue #488 has a separate engine-only browser control. It needs Godot Web export templates and Playwright Chromium. It uses a local Python server and WebSocketPeer, with no Signal Fish service or WebRTC extension:

Bash
python scripts/check-websocket-web-receive.py \
  --godot /path/to/godot --chromium /path/to/chromium \
  --output /tmp/websocket-web-receive

The output directory must be new. The server sends 64 exact 4096-byte text messages and a later sentinel. The script holds polling for 100 ms after each burst command, then checks every delivered byte, type and sequence. Results retain commands, engine/export/source hashes, browser version, poll intervals, every callback and every delivered sequence. This opt-in control preserves engine overflow failures outside the deterministic gates.

Use --credit-bytes 32768 for local consumer body-byte batches, --capacity 524288 for a larger byte buffer, --max-queued-packets 2 to isolate the packet-count boundary, or --hold-ms 1000 for a longer polling hold. Credit uses local observations and adds no network acknowledgement protocol. Zero capacity or packet count keeps the engine default. Keep each control separate from the original uncredited workload.

For a receipt carried on the tested WebSocket, use --credit-transport app-ack --credit-bytes 32768. The receiver acknowledges only the cumulative exact prefix after checking bytes, order and message type. The server releases that many consumed bytes and keeps all receipts. Each complete callback body is 4096 bytes; the byte window covers the entire body. The later sentinel waits for a receipt covering all 64 burst bodies. Use --ack-delay-ms 100 to hold receipts after consumption. The existing local batch control remains the default. This experiment needs a cooperating sender; it adds no delivery guarantee to the SDK or the uncredited workload. The window must also fit the packet-count limit: on a two-packet queue, a 32 KiB window still overflows, while an 8192-byte window admits exactly two bodies. Failed byte or order checks release no receipt or credit. Add --credit-packets 2 to bound outstanding packets directly; the receiver then consumes and acknowledges the same bounded batch.

With Chromium 156 on Linux ARM64, all three supported engines deliver 65/65 exact messages with a 32 KiB application window, 100 ms polling holds and 100 ms delayed receipts. No callback is rejected and outstanding bodies stay within 32768 bytes. The matched uncredited 4.7.2 run delivers 31/65 messages; 34 scoped rejections account for every missing receipt. The later sentinel arrives. These local measurements preserve the original overflow verdict.

On a two-packet queue, the byte-only 32 KiB window still overflows: the 4.7.2 run delivers 2/65 messages with six scoped engine rejections. The same byte window plus --credit-packets 2 delivers 65/65 exact messages with at most two outstanding packets on all three engines, and no callback is rejected. One application window must fit both engine limits; this qualifies the tested local paths, not the uncredited workload or remote networks.

Linux ARM64 and Chromium 156 controls retain these results on Web:

Engine Default buffer 512 KiB buffer 512 KiB, two packets 32 KiB body credit
4.3 Fails Passes Fails Passes
4.4.1 Fails Passes Fails Passes
4.7.2 Fails Passes Fails Passes

Every finite control observes all 65 browser callbacks. The failed byte/count controls retain unique engine rejection identities matching the missing script receipts; the later sentinel is delivered. A larger finite buffer and local credit qualify those controls. The original uncredited receive loss remains open. These are local controls; the remote-path lane below starts the remote qualification.

The receive loss is filed upstream as godotengine/godot#125073. A 2026-11-02 rerun on repository main 1915f97e pinned the boundary on official 4.7.2 Web with default buffers: all 65 browser callbacks arrived, the engine delivered 16 (sequences 0-14 and the sentinel) and logged 49 scoped rejections; the matched 32 KiB application-ACK control delivered 65/65. Upstream review, repair and adoption remain open.

The weekly WebSocket Web Watch CI workflow (Mondays 10:00 UTC, plus on-demand dispatch) keeps this boundary under watch on x86-64. It reproduces the loss on the newest supported official engine, requires the filed signature (the first 15 burst bodies plus the sentinel delivered, 49 scoped rejections) and the green 32 KiB application-ACK control, and fails when the signature moves, flagging either the upstream repair for requalification or a behavior change for triage (#488). A dispatch input watches other official releases, such as a dev snapshot after an upstream fix lands.

Remote path qualification

The WebSocket Remote Path CI workflow drives the same two controls over a real remote network path. A credential-free Cloudflare quick tunnel fronts the fixture server, and the browser reaches it through the Cloudflare edge over TLS, so every burst frame, command and receipt crosses a remote path. The fixture options --port N and --public-origin https://host point the export page and its WebSocket at the public host while the server keeps its local bind; the report records remote_origin.

Four dispatched CI runs (2026-11-03, ubuntu-24.04 x86-64, Chromium 156, runs 41554600823, 41561638882, 41561939096 and 41562318746; Godot 4.7.2-stable on three of them and 4.3-stable on one, one Cloudflare quick tunnel each) settle the lane. The 32 KiB application-ACK control delivered 65/65 exact messages across every tunnel with all 65 browser callbacks and the full ACK window. Every uncredited run reproduced the filed loss at the payload boundary (godotengine/godot#125073) with the three boundary failures, scoped rejections, a complete trace and partial delivery: 16 of 65 (first gap 15 -> 64), 31 of 65 (15 -> 46), 31 of 65 (30 -> 64) and 16 of 65 (15 -> 64). The exact split varies with network segmentation and engine version, so the workflow pins the stable loss class (the three filed boundary failures, scoped rejections with a complete trace, sub-burst delivery). The weekly schedule (Mondays 12:00 UTC) follows the other watches; dispatch runs collect extra path samples on demand. A same-day pre-merge devcontainer run on Godot 4.3 Web also gave the local 16-of-65 counts.

The same workflow fronts the paced native receive suites (check-websocket-native-receive.py) with the same tunnel. The driver options --port N --public-origin https://host pass the fixture options --port=N and --public-origin=https://host to every selected case: the fixture server binds the tunneled port and the engine connects wss:// at the public host. Result records from tunnel runs carry remote_origin. The lane runs controls and filed defects for each receive issue class (#491 fragmented assembly around control frames, #492 peer reuse retention, #496 full-buffer empty-fragment stall and #498 Unicode close reasons across reads). Controls must pass over the remote path, with the driver's full per-case verdict. Defect records must be complete, paced and cleanly exited. Each defect verdict is pinned per engine to the signature the collected runs reproduced: the gate keeps a map keyed by engine major.minor. A pinned defect that passes or fails differently means the path masked its trigger or upstream moved; requalify the filed issue and update the map. Dispatching an engine without a map records its defect outcomes as data; collect runs, then add its map.

Coverage by filed issue, from the same suite on each engine:

Issue Filed symptom Official-engine remote watch Repaired-engine remote lane
#491 fragmented bytes change around Ping/Pong (exact receive prefix) fails on 4.4.1-4.8-beta2; passes on buffered 4.3, which predates the regression passes
#492 a fresh peer returns an empty packet after CLOSING reuse (fresh exact bytes) fails on 4.4.1-4.8-beta2; passes on buffered 4.3 passes
#496 a full buffer plus an empty final fragment stalls (complete message missing) fails on 4.4.1-4.8-beta2; passes on buffered 4.3 passes
#498 a valid Unicode close reason becomes 1007 (exact close code, reason and empty packet queue; matching client close frame) fails on every collected engine, including 4.3 and 4.8-beta2 passes

The 4.3 entry in the #496 row is the paced remote case; the standalone full-buffer reproduction still drops the following packet on official 4.3. The repaired legs are the 4.3 backport, 4.4.1, 4.5.2, 4.6.3 and 4.7.2 on Linux x86-64, plus 4.4.1, 4.5.2, 4.6.3 and 4.7.2 on Windows x86-64 under Wine and 4.7.2 on a native Windows runner. Linux legs run at the 3-byte read cap except the cited uncapped 4.7.2 run; Wine and native Windows legs run uncapped. The same ladder qualifies the #485 receive repair. The separate Gameplay Shutdown Remote lane covers active-shutdown close semantics over a remote path on official Web 4.3, 4.4.1 and 4.7.2, native 4.4.1 and 4.7.2, and the pinned repaired 4.3 native backport. Each issue stays open for upstream adoption (resubmission tracked in #536). The macOS map's pinned gaps cover the #492 and #498 families only. Targets with no collected map (bare-metal x86-64 and engine families not yet collected) also stay open.

A 2026-11-03 pre-merge devcontainer run (official 4.7.2-stable native on Linux ARM64, one Cloudflare quick tunnel, ten selected cases) matched the local 4.7.2 classification exactly: all four controls passed over the tunnel, including the 65-packet burst control, and all six defects failed with their filed signatures. The tunnel runs exposed two fixture races. The pending #492 continuation could flush to the old tunnel connection before the replacement dial, so the retirement check lost its precondition; the fixture now holds the queued continuation until the replacement retires it, which makes the check deterministic on any path (a post-fix tunnel run fails the #492 defect with the filed signature alone). The limit fixture's pong check now waits within the phase budget before judging, because the pong can still be in flight on a remote path when the message completes. The #498 maximum-length mixed close reason also hit a paced-write failure when the engine aborted mid-write in one tunnel run; the gate rejects paced-write errors as unclean samples.

Seven dispatched CI runs (2026-11-03 and 2026-11-04, ubuntu-24.04 x86-64, official editor binaries, one Cloudflare quick tunnel per native job) pinned the native map. On 4.7.2 (runs 41591258643 and 41684196081), 4.6.3 (run 41937707094), 4.5.2 (run 41937461420) and 4.4.1 (run 41684522769, whose native job is green while the run is overall cancelled after its Web job lost a concurrency race) all four controls passed over every tunnel, with the 65-packet burst control delivering every packet (largest delivery gap 283 ms against the 3000 ms batch deadline). All six defects failed with their filed signatures, apart from the pre-#541 retirement race in the first 4.7.2 run noted below. The gate now carries a map for every stable family from 4.3 to 4.7 and for the 4.8 beta family. The retired-continuation failure in the first 4.7.2 run was the fixture race #541 removed; post-fix runs fail the #492 reuse defect with fresh exact bytes alone and retire exactly 2 queued bytes. On 4.3 (runs 41684912691 and 41685515285) the #491/#496/#492 shapes pass: official 4.3 predates those regressions. The #498 close shapes fail with the filed signature on every collected engine. Same-day devcontainer runs on Linux ARM64 (4.7.2) matched the CI classification. The weekly schedule (Mondays 12:00 UTC) runs both lanes; dispatch runs collect extra path samples on demand and add maps for new engines.

Godot 4.8-beta2 is the first collected pre-release family (run 42108167042, 2026-11-05, ubuntu-24.04 x86-64, official editor, one Cloudflare quick tunnel per job). Both lanes follow the 4.4-4.7 shape. Native: the four controls pass, including the 65-packet burst, and the six defects fail with their pinned signatures, so the gate carries a 4.8 map. Web: the 32 KiB application ACK control delivers 65/65 with the full 64-packet ACK window, while the uncredited run delivers 16/65 with 49 scoped rejections (first gap 15 -> 64), the same loss class as godotengine/godot#125073. The installer fetches pre-release builds from godotengine/godot-builds; stable releases keep using the main repository.

The WebSocket Native x64 lane extends the same tunnel idea to the experimental source-build repair (#485). With remote_path on it builds the pinned 4.7.2-stable source twice on ubuntu-24.04 x86-64, repaired via scripts/repair-websocket-stable.py and unpatched, enables the mbedtls module so the engines can speak wss://, and fronts the fixture server with a Cloudflare quick tunnel. The repaired engine runs the ten pinned receive cases over the tunnel at the lane's forced 3-byte read cap on the default Linux dispatch; the baseline runs uncapped, the conditions the official-engine pinned signatures were collected under. The repaired engine must pass every case. The baseline must keep its pinned map: each expected-failure case must keep failing with its filed signature, and each expected-pass case must keep passing, so neither a pass nor a drift can come from the path masking a trigger. The pinned 4.3 map expects only its two #498 close shapes to fail; official 4.3 predates the other four defect shapes, and the gate requires those four to keep passing on the 4.3 baseline. That requirement applies to a Linux engine family with a pinned remote-path map; Windows and unpinned families record defect outcomes as collection data. The gate checks binary identity by SHA-256, tunnel provenance, paced writes, clean exits and per-case verdicts, keyed by engine family for the baseline map. A baseline defect sample may contain engine errors (for example the #492 reuse case's PacketBuffer ERR_BUG line); the exit, paced-write, record and pinned-signature checks still bound it. Pass-required cases, every repaired case and each baseline control, must stay free of engine errors, which the driver's verdict already enforces.

Three dispatched runs (2026-11-03, 1d2e7604, official 4.7.2-stable importer, one quick tunnel per variant) settle the first remote-network qualification of the repair. In the capped run (41700349763) the repaired engine (4b58fac581f2b2f293d85b2eb313bf59d9fc227a9d121f3fbaeb21bfd7ae6130) passes all ten cases over the tunnel with every read capped at 3 bytes; the 65-packet burst control delivers every packet with a largest delivery gap of 255 ms against the 3000 ms batch deadline. The unpatched baseline (77bbcc386e6415a611d827051d24edd82b7eb9f44fb858ab57a4413405c8144e) from the same source and flags passes its four controls and fails all six defect shapes with the exact pinned signatures on its own tunnel. In the uncapped run (41700353810) the repaired engine passes all ten cases uncapped (burst largest gap 276 ms). Its baseline run reproduced five of six pinned signatures exactly and then hit the documented mid-write abort on the maximum-length mixed close shape (paced socket write failed); the gate rejects paced-write errors as unclean samples, and the re-dispatch (41706656767) collected the clean sample: the baseline fails all six pinned shapes again. The abort stays a documented one-off, as in the official-engine lane. The first dispatch attempt (41697078313) failed every remote connect with WSS is not available in this build: the minimal source build shipped no TLS module. Its repaired engine still passed the full local paced suite, 184 of 184 cases at the 3-byte cap. Dispatch runs collect extra remote samples on demand; the cap applies on the default Linux dispatch, so a Windows or zero-cap remote dispatch qualifies the repaired leg uncapped.

The pinned 4.3-stable backport 77dcf97d82cbfe4e4615475fa52ca03da645dbd8 completes the same remote leg (run 41734479410). The repaired engine (8aebdbce741747e08ba00ec694764e38d9d311ee94e5193d582c447197d831f3) passes all ten pinned cases over its Cloudflare tunnel with every read capped at 3 bytes; the 65-packet burst control delivers every packet with a largest delivery gap of 248 ms against the 3000 ms batch deadline. The unpatched 4.3 baseline (68c4bce9bca9c3947428069ab1d30ad5c2c2ebd6ee5c7acbef9749f293fc70df) on its own tunnel passes its four controls and the four pre-regression shapes, and fails exactly the two #498 close shapes with the pinned close signature, matching the official-engine 4.3 watch maps. That closes the 4.3-backport leg of the #485 remote-network qualification.

The pinned 4.4.1-stable repair 49a5bc7b616bd04689a2c89e89bda41f50241464 completes its Linux x86-64 remote leg (run 41754464659). The repaired engine (7531c11dc69503e334d9048d6dbdb873f920942988a0d6fa58d95866ca8ec900) passes all ten pinned cases over its Cloudflare tunnel with every read capped at 3 bytes; the burst control delivers 65 of 65 packets with a largest delivery gap of 248 ms against the 3000 ms batch deadline. The unpatched baseline (e082eb387fa1f3e008c2967712fc644a770e31ad99bd6f363eb368f16aadb5dd) on its own tunnel passes its four controls, fails all six pinned defect signatures, and delivers its burst 65/65 (largest gap 262 ms). A first dispatch (41751187507) had already qualified the repaired leg; its baseline sample stayed unclean (the documented mid-write abort) while the lane still rejected engine-error defect samples outright, so the gate now tolerates a defect's filed symptom and the re-dispatch collected the clean sample above.

The pinned 4.5.2-stable repair 6ce3de25aa58466e14ef354703ba8d9791a417da completes the same leg (run 41754468768). The repaired engine (f158c982bff211daf0c41e197d79f90b8f09acddf99f0f55d0015162ac6ce50f) passes all ten cases at the 3-byte cap (burst 65/65, largest gap 255 ms). The unpatched baseline (4defc7d31eab80f57ee4abb07ba09ce90b2ef6a23df079ca62e3980ba1a8151f) passes its four controls and fails all six pinned signatures on its own tunnel, so the lane pins the 4.5 family map to those six signatures.

The pinned 4.6.3-stable repair 35e80b3a8822a9df9be390814b62f44c0a9c69e8 completes the same leg (run 41754472264). The repaired engine (ce17e58543d06978d479d7568bd561713f022cdbafc85948f5cbef37a7fe5210) passes all ten cases at the 3-byte cap (burst 65/65, largest gap 269 ms). The unpatched baseline (7ab2a67a48b4266743d2e05acf0a4db0551deba2117f8e7bf55e6dd7b2af1714) matches the 4.5 baseline outcome set and the lane pins the 4.6 family map the same way. That closes the 4.4.¼.5.2/4.6.3 stable-repair remote legs of the #485 remote-network qualification.

The Windows dispatch closes the last planned #485 remote leg. In the first run (41754475885) the repaired 4.4.1 engine (3e7f09f711eac24df914e549762a92e1cf10127f6c096bf0b555f506ac949e6f) passes all ten pinned cases over its Cloudflare tunnel uncapped under Wine, with the burst control at 65/65 packets (largest gap 257 ms). The unpatched Wine baseline (d5b78c0fe2940d0874fb0b488303a88157192fbb5858b93ceafa91e33eb1d75b) records clean outcomes on nine of its ten case samples (four controls pass, five defect shapes fail with the pinned signatures) but hits the documented mid-write abort on close-3-mixed-server, so the gate rejects the sample and a re-dispatch (41763583302) collects the clean one: the repaired engine (42cb1f63f4ae64b47e9487a1246e5d57bf51a1081a7bbfa411ad605f25d8c7c1) passes all ten cases again (burst 65/65, largest gap 248 ms) and the baseline (fe39d6ccf1f79285f45afc0883953c3bf631c50165348f0c10fc6b430c4ab9f4) passes its four controls and fails all six pinned signatures, its close-3-mixed-server sample now clean. The second collected Wine sample (41774059122) confirms the map: the repaired engine (4905f4af3ea7c005a82aed0ddeddb9af8c91a72aa2a508a9ae138ee60832bd19) passes all ten cases again (bursts 65/65, largest gap 283 ms) and the baseline (38c0e2580628c19117bf47dc9ca9cec7b56883012bff3882bb626b583e5beb45) passes its four controls and fails the same six pinned signatures with clean close samples (bursts 65/65, largest gap 909 ms against the 3000 ms batch deadline, below the recorded 1.14 s benign Wine stall). The gate pins the Wine 4.4 family map from these two runs; the other families stay collection data on Wine until their own collected runs pin maps.

The test-only observer wraps the engine's assigned browser onmessage handler. Each rejection records the physical connection, callback ordinal, byte length, bounded prefix, rejection reason and browser clock while that handler is active. Payload-capacity and packet-count rejections stay distinct. An error outside the handler has no callback identity. Callback arrival alone does not prove engine acceptance. A complete trace must account for omitted records before comparing rejected identities with script receipts.

The active-gameplay driver also accepts these comparison controls:

  • --relay-packets-per-second 240: use elapsed time to set the offered target, with at most 16 sends per frame to catch up. Zero keeps the original frame rate.
  • --relay-credit-bytes 32768: cap outstanding 4096-byte bodies using the other local seat's receive count. JSON framing and other protocol traffic add bytes.
  • --relay-credit-packets 2: cap outstanding relay packets the same way. Combine it with a byte window so the application contract fits both engine limits.
  • --receive-pause-ms 100: hold engine polling after each mesh generation opens. Other application and mesh work continues. Zero leaves polling unchanged.
  • --inbound-buffer-bytes and --max-queued-packets: set the engine capacities. Zero preserves each engine's default. A Web property value does not establish the browser's backing queue capacity.

Add --relay-credit-transport app-ack --relay-credit-bytes 32768 for an application ACK comparison. Each receiver sends a cumulative exact-prefix receipt through GameData. Only a validated receipt from the current remote player, room and gameplay generation advances the sender's byte window. A missing or corrupt data packet stops new ACK progress. The default local credit mode keeps the shared-process receipt counter. ACK records retain physical connections, sent/received prefixes and window peaks. A packet window from --relay-credit-packets is enforced on both seats and recorded beside the byte-window peaks.

Application ACKs add protocol traffic and change the transmitted schedule. This is an opt-in drill control, with unchanged 4096-byte data bodies. It establishes no SDK-wide delivery guarantee. Preserve the original uncredited failures and qualify real remote paths separately.

On Linux ARM64, the 32 KiB application ACK control with a two-packet window passes the Web 4.7.2 stalled-close case against the released v0.12.0 server with 50 ms TCP delay, 127-byte proxy writes and a 100 ms initial polling hold. Both seats peak at two outstanding packets, keep exact relay prefixes, exact mesh receipts in all three modes and the required nonresponding close outcome.

On Linux ARM64, the 32 KiB application ACK control passes all three shutdown cases on native Godot 4.3 and Web Godot 4.4.¼.7.2 with 50 ms TCP delay, 127-byte proxy writes and a 100 ms initial polling hold. All exact mesh, cached retirement and fresh-gameplay checks pass. A matched uncredited Web run retains a scoped rejected callback and relay gap from 151 to 152; both seats still preserve the semantic shutdown close. The engine receive defect remains open. The application ACK control qualifies its bounded local workload.

Logs identify the seat and physical connection, reported capacities, poll intervals and duration, queued packet peaks, total received frame bytes, first relay gap, offered target, credit blocks from either window, and peak outstanding body bytes and packets. All original shutdown, cached-packet and exact mesh checks remain required. Keep rate, credit, pause and capacity changes separate in reports.

A 240-packet/s native 4.3 run with a 100 ms poll hold fails on an engine overflow and a first delivered gap from 15 to 24. Its matched 32 KiB body-credit control passes all gameplay checks. The credit result qualifies that local control; it leaves the original uncredited workload failed. A new six-case Godot 4.4.1 native/Web cohort passes graceful shutdown, reset, nonresponding close, cached metadata, retirement and fresh mesh recovery at the original four-per-frame rate. Three new Godot 4.3 Web shutdown controls also pass, and the 4.3 Web shutdown leg is now qualified over the remote path. Godot 4.3 native receive qualification remains open.

Delayed relay shutdown

Server v0.12.0 release qualification

The published Linux ARM64 server v0.12.0 archive was checked against its published SHA-256. Its source revision is 92842406. The server executable SHA-256 is 90d8cd0799ee0e61e20cfbf2da8e5996811c100f7b794560bdd48676fac110d1.

With 50 ms TCP delay, 127-byte proxy writes, a 100 ms polling hold and the original four 4096-byte bodies per frame:

  • Repaired native Godot 4.3 passes graceful, reset and stalled-close recovery. Every mode preserves exact relay prefixes, cached mesh retirement and fresh mesh gameplay. An earlier graceful attempt exhausted the native send queue and lost the semantic close. Issue #510 adds recoverable ERR_BUSY handling.
  • Repaired native Godot 4.7.2 also passes all three shutdown modes.
  • Official Web Godot 4.7.2 on Chromium 156.0.8078.4 passes graceful shutdown. Its reset and stalled-close runs retain interior relay gaps. Browser socket 2 rejects sequences 56-59 in four 4229-byte callbacks; socket 4 rejects 34-35. These are scoped engine callback rejections. Issue #488 remains open.
  • The matched Web application ACK control (--relay-credit-transport app-ack --relay-credit-bytes 32768) passes all three modes with no callback rejection. This window counts body bytes. Other protocol traffic adds bytes.

Those Web rejection captures preceded the native send-capacity change. A later cohort passes all three Web modes with and without ACK credit. After the final native capacity update, another uncredited stalled-close control again fails: socket 1 rejects five 4229-byte callbacks for sequences 35-39, and delivery jumps from 35 to 40. Graceful and reset controls pass in that cohort. Passing schedules leave these failures unresolved. The native capacity change does not repair Web receive queues.

Reports separate relay and ACK busy attempts. The drill checks that both stop after terminal state. During a valid observed close it permits one application refusal and one separate transport-status refusal. Close metadata, delivered prefixes, retirement and recovery remain strict.

Join-only admission also passes on official Godot 4.3, 4.4.1 and 4.7.2 for both protocol versions. Each native engine passes the real WebSocket smoke with a 16-byte outbound limit: oversized UTF-8 text and binary frames return ERR_INVALID_DATA, an exact-capacity frame is delivered, and the close completes.

All attempts, commands, hashes and reports are retained locally under /tmp/session470/ and progress/session470-artifacts/. Setup failures and attempts that imported intermediate source remain separate. These are local Linux ARM64 controls; remote networks, other targets and upstream engine adoption remain open for #482, #485 and #488.

Delay controls

Add --network-delay-ms 50 to the active-shutdown command to delay each TCP direction by 50 ms. This includes relay data and signaling. Mesh UDP and HTTP control stay local. The workload, close deadlines and exact gameplay checks stay unchanged. This is local network emulation; real remote paths need their own qualification.

Use --network-chunk-bytes 127 to bound each proxy read and write. TCP may coalesce these writes before Godot reads them. This does not promise a matching engine read size or WebSocket frame fragmentation. Queue slots use a 1 MiB payload budget per direction, capped at 8192 chunks. Reports retain the effective byte/count bounds, bytes, chunk counts, peak queue size and observed delay for each physical connection. A full queue can add TCP backpressure. EOF follows queued data. Socket errors and cleanup cancellations stay visible. Each passing case must forward both directions on at least four connections and meet the requested minimum delay.

The proxy's loopback integration tests are opt-in: PYTHONPATH=tests python -m unittest test_delayed_tcp_proxy. They cover large exact binary streams, half-close replies, actual delay, reset and idle teardown.

Browser logs include socket state reads and close callbacks. On the original 4.4.1 delayed run, both browser sockets were CLOSING while the engine reported OPEN. A send returned FAILED; the SDK discarded the peer before close 4000 arrived about 80 ms later. The transport now refuses further sends and polls for that callback for at most one second, then falls back to one failure. Text and binary refusals use the same rule. All strict receipt, shutdown, cached metadata, retirement and fresh-gameplay checks remain required.

One repaired 4.4.1 native/Web 50 ms cohort passes all six shutdown cases. Native 4.3 still fails exact relay receipts in all three delayed controls; its matched 32 KiB local body-credit controls pass. The 4.7.2 small-write cohort preserves shutdown and mesh checks, but its Web graceful case drops relay packets after restart (#488). Both seats expect sequence 24 and receive 26/27 after engine buffer errors. Its matched 32 KiB body-credit cohort passes all three cases. Later final-source 4.4.1 Web graceful and reset runs also fail on receive overflow. The graceful gap is from sequence 535 to 536 during the first generation; its close, retirement and fresh mesh remain exact. Credit changes the local offered work; it does not repair or qualify the original uncredited receive boundary. Every failure stays retained.

Add --trace-receive to retain scoped browser callback and rejection records for the active-shutdown drill. The driver freezes the observer source per cohort, copies it per case, and records its hash. Generated relay bodies also get a parsed relay_sequence; other messages leave it null. Snapshots are collected before the browser closes, including failed controls.

A new uncredited 4.7.2 Web run with 50 ms delay and 127-byte proxy reads fails with a scoped rejection and a generation-one relay gap from 21 to 22. A separate 100 ms polling-hold control identifies rejected relay sequence 23 and the matching application gap from 23 to 24. Both retain GoingAway, one close 4000/server_shutdown per seat, retry eligibility, cached retirement and exact fresh mesh receipts. Their overall verdicts stay failed. Matched 32 KiB body-credit controls pass. Passing attempts and every failure stay separate; this evidence leaves engine repair and remote qualification open.

Controlled lifecycle publication

tests/smoke/run_controlled_lifecycle_acceptance.gd adds local server barriers for #356, #358 and #367. Use an isolated server checkout at 182cb234c7aa601be3c2aad881948b3c5b853aaa, the config above, and Rust 1.94.0. Apply only tests/fixtures/controlled_lifecycle.patch for this drill.

Bash
# From the isolated server checkout:
git apply /path/to/signal-fish-client-godot/tests/fixtures/controlled_lifecycle.patch
cargo build --locked --bin signal-fish-server
export SF_GODOT_BARRIER_DIR="$(mktemp -d)"
printf '%s\n' "$SF_GODOT_BARRIER_DIR"
SF_GODOT_BARRIER_DIR="$SF_GODOT_BARRIER_DIR" target/debug/signal-fish-server
# In another terminal, from the Godot client checkout, set the printed path:
export SF_GODOT_BARRIER_DIR=/path/printed/above
godot --headless --path . --script tests/smoke/run_controlled_lifecycle_acceptance.gd

Run one drill per server and barrier directory. The server consumes .arm files, writes .reached files and waits for .release files. Each server wait has a 15-second limit. Each client wait has an eight-second limit and the full drill has a watchdog. Client cleanup releases held publication barriers. After testing, stop the server, remove the barrier directory and reverse the patch with git apply -R. Rebuild before using that checkout for ordinary server runs. The patch is acceptance instrumentation.

The drill checks these new boundaries on Godot 4.3 Linux arm64:

  • 367: two real reports from one socket call the public server dispatcher

    concurrently. The first pauses after state acceptance and before fanout. The second reaches the report lock before release. Both true/false orders reach the v3 observer in acceptance order; v2 receives no transport events. Server assertions under the report lock check stored state at each queue admission. A duplicate is suppressed. Both observers receive later gameplay. The ordinary transport-report drill above covers reconnect generations.
  • 358: a Direct metadata transaction pauses inside the commit hook before

    storing the endpoint. Join and leave each reach the room mutation lane while it is held. No plan or membership change appears before release. Clients converge on the fresh endpoint and generation; plan callbacks see peers in the current roster, and relay gameplay succeeds afterward.
  • 356: the first registration attempt returns failure at the existing result

    seam. A successful retry preserves terminal identity, reason, roster removal, replay and fresh gameplay. A panic on retry exhausts publication and closes the viewer once with 1000/unregistered. No terminal acknowledgement is sent through the unregistered generation. Room members still receive one removal. Fresh authentication and spectator admission restore gameplay. Both recovery and exhaustion run with v2 and v3 viewers.

The server's normal WebSocket reader dispatches serially. The fixture starts transport-report tasks through the public library entry point to reproduce embedded-host concurrency. It preserves the production report, lifecycle, room and queue gates. The injected registration failures qualify recovery behavior at that seam; the drill does not establish a request-triggered coordinator failure. The publication drill below adds authoritative spectator removal, correlated result decoding and controlled Direct reconnect publication. Browser interop is separate.

Gameplay publication boundaries

Using the same patched server and barrier directory, run:

Bash
godot --headless --path . --script tests/smoke/run_publication_boundary_acceptance.gd

This drill passed with Godot 4.3 Linux arm64, client base 5abec3c and server 182cb234 plus the checked-in controlled_lifecycle.patch:

  • 368: pause spectator admission before baseline registration while holding

    the room mutation lane. A player join reaches that lane and waits. Gameplay continues to an incumbent player. The new viewer receives no traffic before publication. After release, it receives one baseline then one player delta, a complete roster and fresh gameplay. Prior gameplay is absent from the new viewer's history. The drill covers v2/v3 JSON and raw MessagePack.
  • 356: an administrative detach with reason removed recovers after the

    first registration failure. v2/v3 viewers receive one terminal with the original room identity and reason. A connected member receives one removal; an absent v3 player receives that removal once in decoded replay. Replay requires consumer dispatch. Readmission and gameplay succeed.
  • 356: a fixture sends one raw v3 RoomOperation/LeaveSpectator after

    negotiating room_operation_ids. Recovery preserves operation identity, SpectatorLeft payload, room identity, reason and roster. The SDK emits one informational result and leaves its cached session unchanged. The server removes the viewer from routing. This fixture adds no room-operation API or admission support. The ordinary SDK uses leave_spectator().
  • 358: pause a Direct endpoint commit while reconnect reaches the mutation

    lane. Two reconnect cycles receive their baseline before fresh plans, rotate the credential and converge on the new endpoint and generation. A game-owned StreamPeerTCP connects to a real loopback listener. It sends payloads after reconnect and running endpoint refresh. Service disconnect, endpoint replacement and Relay reset close the prior socket; the listener observes EOF. Relay gameplay continues after teardown.

The added barriers are spectator-baseline and reconnect-enter. spectator-remove.arm makes the next spectating client's Ping call the existing administrative detach path with reason removed, then process the Ping normally. Combine it with detach-error.arm for recovery. This trigger is local acceptance instrumentation; it is absent from ordinary builds. The mutation, routing, registration retry and publication paths remain intact. The drill releases held barriers and closes application sockets on failure. All waits and the overall watchdog retain the bounds described above.

The pressure drill below covers server delivery pressure. The roster recovery drill covers observer metadata. This native TCP fixture provides no browser Direct-socket contract; browsers require supported transports.

Server delivery pressure

tests/smoke/run_delivery_pressure_acceptance.gd covers #368 at the server outbound queue. Use the isolated server, config and toolchain above. Apply controlled_lifecycle.patch, then tests/fixtures/delivery_pressure.patch. Set websocket.send_queue_capacity to 4 and keep websocket.slow_consumer_timeout_ms at 5000. Rebuild the server and start it with a fresh SF_GODOT_BARRIER_DIR, shared with the client process:

Bash
godot --headless --path . --script tests/smoke/run_delivery_pressure_acceptance.gd

The added barrier is writer-<spectator-id>. After writing a control frame, the server writer waits before selecting another queue entry. The client arms only its spectator's writer and wakes it with Ping. The sender's Pong then proves that the server processed every ingress frame while the writer was held. The production queue admission, coalescing and reporting paths remain active. The drill releases the writer on failure. Reverse both patches and restore the queue config after testing.

  • v3 JSON latest: eight updates to one key produce the final value, exact latest_superseded ranges for sequences 1-7 and matching typed counters.
  • v3 JSON volatile: a full four-entry queue evicts sequences 1-4 and delivers the newest four values in order. Exact volatile_dropped ranges and typed counters match. Both lossy cases publish the report before surviving data, preserve the spectator role and receive fresh reliable gameplay afterward.
  • Reliable: five frames fill the four-entry queue and trigger one spectator close with code 4002. The sender stays in its room. The viewer clears its state, authenticates on a fresh connection, rejoins and receives fresh data. The matrix covers v2/v3 JSON and raw/decoded MessagePack, plus v3 opaque rkyv/protobuf. Opaque test bytes remain application-owned.

Latest and volatile sends use a raw v3 text fixture. The public SDK send methods currently use reliable delivery. This drill adds no lossy-send API or binary delivery-metadata API. The v2 verdict applies to the pinned server build; v2 has no spectator_relay capability token.

Roster recovery contract

tests/smoke/run_roster_recovery_acceptance.gd is an opt-in regression for issue #374. It needs the ordinary fixed server above, with no instrumentation:

Bash
godot --headless --path . --script tests/smoke/run_roster_recovery_acceptance.gd

Use server b264ec7 with the ordinary config above. Godot 4.3 Linux arm64 passed six cases: v2/v3 observers with JSON and raw/decoded MessagePack. A seated observer, an incumbent spectator and a spectator admitted during absence restore the current name, readiness and authority before each reconnect callback. Each case includes two reconnect cycles, exact resumed gameplay, spectator role and terminal leave. Reconnect emits no synthetic player join. Success prints LIVE_SERVER_ACCEPTANCE_PASS.

Server 182cb234 omits the metadata and fails this drill. The new single-frame contract shipped in server PR #1169. The deterministic client suite also covers missing/malformed metadata, ID/epoch/sequence matching, duplicates, older epochs, replay baseline precedence, room reset, bounded roster/departure history and mesh incarnation retirement.

Expired reconnect cleanup

tests/smoke/run_expired_cleanup_acceptance.gd covers issue #357. In an isolated server checkout at b264ec7, use the config above with server.reconnection_window set to 1. Apply this repository's tests/fixtures/expired_cleanup.patch, then rebuild:

Bash
# From the isolated server checkout:
git apply /path/to/signal-fish-client-godot/tests/fixtures/expired_cleanup.patch
cargo build --locked --bin signal-fish-server
export SF_GODOT_EXPIRY_DIR="$(mktemp -d)"
SF_GODOT_EXPIRY_DIR="$SF_GODOT_EXPIRY_DIR" target/debug/signal-fish-server

In another terminal, set SF_GODOT_EXPIRY_DIR to the same directory and run:

Bash
godot --headless --path . --script tests/smoke/run_expired_cleanup_acceptance.gd

A Ping with prepare.arm captures the server regression's controlled residue: the server removes a seat while retaining its socket, registers a pending reconnect, waits for expiry and snapshots its real cleanup candidate. It then sends RoomLeft and Pong. The client joins the same room with the same socket identity and receives a new RoomJoined credential. A Ping with cleanup.arm runs the saved candidate through the production cleanup method before Pong. The cleaned file reports zero removed seats.

Godot 4.3 Linux arm64 passed four cases with v2/v3 observers and JSON/decoded MessagePack. The fresh seat becomes ready, sends exact gameplay and reconnects with its current credential. An ordinary expiry control refuses the expired membership and permits a fresh login/join. The actor uses v3 because this server pre-issues reconnect credentials on v3. V2 observer results qualify this tested server build. They do not imply v2 credential issuance.

The fixture forces a server-side residue seam. It does not establish that ordinary remote messages can create the stale record. Candidate capture has a four-second deadline; client phases wait eight seconds, and the runner has a two-minute watchdog. Run one drill per server and directory. Stop the server, remove the directory, restore its config and reverse the patch after testing. The fixture is local acceptance instrumentation.