Deterministic Testing¶
The test suite has no framework dependency and no network. Custom
SceneTree runners under tests/ run with godot --headless --script and
produce deterministic results:
- Byte-pinned fixtures. Codec tests compare exact bytes against fixture files.
- Injected clocks. Reconnect backoff tests step time manually. No sleeps, no wall-clock waiting.
- Synchronous fake transport.
SFFakeTransportis an in-memory test double with injectors (inject_open(),inject_text(),inject_server_message(),inject_binary(),inject_close(),inject_failure()) and recordedsent_text/sent_binarybuffers.
Suites¶
- Protocol fixtures. The upstream v2 samples (server v0.9.2) are
vendored byte-identically under
tests/fixtures/upstream/, and the codec is pinned to them. Hand-built fixtures cover all 24 server message variants plus malformed input. - Transport. The
WebSocketPeeradapter runs against fakes for connect, receive, send, close, error, and backpressure behavior. - Client. Connect, authenticate, join, game data, authority, spectators, reconnection, and the WebRTC mesh all run on fake transports and injected clocks.
Run locally¶
all runs the private-helper guard, GDScript and Python format/lint checks,
strict Python typing, the Prettier and markdownlint checks, and the Godot
suites. Subcommands for targeted runs:
static: private-helper guard, format/lint, and Python types. No Godot install.python-types: Python format, lint, and strict types.private-helpers: the private-helper static guard.format: gdformat checks.lint: gdlint.prettier: the Prettier half of CI's Source formatting job. Needsnpm ci --ignore-scriptsfirst.markdownlint: the markdownlint job of CI's Docs Validation. Samenpm cirequirement. To autofix, runpython3 scripts/check-source.py markdownlint write. The pin matches the version bundled by themarkdownlint-cli2-actionin docs-validation.yml - when bumping that action, re-pinpackage.jsonto its bundled version.godot: the SceneTree suites.
Smoke (opt-in)¶
smoke is never part of all. It drives a real WebSocketPeer round-trip
against a local RFC 6455 test server: open, echo round-trips, close
handshakes in both directions, and refused-dial failure.
Type strictness¶
Every GDScript warning class the engine matrix registers is pinned to error
level in project.godot except three: := type inference and discarded
return values stay ignorable by style, and the 4.7-only missing_await
stays unpinned because a marker naming it is a parse error on engines that
lack the class. The checks apply to the addon as well:
the engine default exclude_addons is explicitly disabled, so a warning
regression fails CI anywhere in the repo. The small set of statements that
are not yet fully typed carry an explicit @warning_ignore marker naming
exactly what they trigger: grep for @warning_ignore in addons/ for the
live list.
Live server acceptance (opt-in)¶
Join-only room admission¶
With an isolated server v0.10.0 or later, run:
godot --headless --path . --script tests/smoke/run_join_only_acceptance.gd -- ws://127.0.0.1:3536
For v2 and v3, two join-only requests must each return ROOM_NOT_FOUND for
the same absent room. An explicit false then creates that room. An explicit
true joins it, and both players start a game and exchange exact gameplay.
The deterministic suites also reject invalid flags and a missing room code.
Spectator and session acceptance¶
tests/smoke/run_live_server_acceptance.gd drives real Godot clients through
SFWebSocketTransport. Use an isolated local server. The drill creates rooms
and changes their membership. It requires a server with spectator_relay
and Direct endpoint refresh support.
The measured server source is
182cb234.
Build it with its pinned Rust toolchain:
git clone https://github.com/Ambiguous-Interactive/signal-fish-server.git
cd signal-fish-server
git checkout 182cb234c7aa601be3c2aad881948b3c5b853aaa
cargo build --locked --bin signal-fish-server
cp config.example.json config.json
python3 - <<'PY'
import json
from pathlib import Path
path = Path("config.json")
config = json.loads(path.read_text())
config["session"]["default_topology"] = "host"
config["session"]["enable_webrtc"] = False
config["protocol"]["enable_rkyv_game_data"] = True
config["protocol"]["enable_protobuf_game_data"] = True
config["rate_limit"]["max_room_creations"] = 1000
config["rate_limit"]["max_join_attempts"] = 1000
path.write_text(json.dumps(config))
PY
./target/debug/signal-fish-server
From this client repository, in a second terminal:
An optional argument after -- selects the server base URL, for example
-- ws://127.0.0.1:3536. Success prints LIVE_SERVER_ACCEPTANCE_PASS and
exits zero. Each wait has an eight-second deadline and the drill has a
two-minute watchdog. A failed phase exits nonzero.
Godot 4.3 on Linux arm64 passed these live checks:
- Spectator baseline state is installed before its signal. Live player join/leave, lobby readiness, GameStarting and gameplay preserve spectator role. Voluntary leave clears the room and returns to authenticated state.
- Gameplay reaches spectators with the player's sender ID: v2/v3 JSON, MessagePack raw bytes and decoded values, and opt-in v3 rkyv/protobuf bytes. Raw binary signals carry the negotiated encoding.
- Two Direct clients receive each fresh shared plan after a host endpoint change, endpoint loss with a successor, and loss of the final endpoint. The Relay reset clears host, endpoint, peers and ICE state. Relay gameplay still arrives. Duplicate and non-host metadata keep the current generation; Pong replies provide processing barriers on both connections.
Additional drills use the same server setup:
godot --headless --path . --script tests/smoke/run_room_transition_acceptance.gd
godot --headless --path . --script tests/smoke/run_live_transport_reports.gd
The room transition drill queues 80 real receive packets before dispatch, then pipelines spectator leave and join on one connection. It drains one packet per poll and checks terminal/baseline order, payload order and sender, room identity, new roster, and fresh gameplay after rejoining the first room. It covers v2/v3 JSON and raw MessagePack. It also overlaps Direct host endpoint updates with another player's join and leave. Current players converge on one fresh endpoint and generation; each observed plan uses the current roster. These overlapping socket requests do not force internal server race barriers.
The transport report drill uses a v3 reporter, a relay-only v3 observer, and a v2 observer. It checks both state-transition orders, duplicate suppression, self/v2 exclusion, and fresh report state after reconnect. Gameplay on the same connection provides a processing barrier and confirms relay delivery. It covers serial WebSocket dispatch. The controlled lifecycle drill below covers concurrent embedded-host dispatch. On the older pinned server, this drill exposed the missing peer metadata. The roster recovery drill qualifies the repair.
All three drills passed on Godot 4.3 Linux arm64 against the pinned server above. They accept the same optional base URL. Each phase has an eight-second deadline; the room transition watchdog is two minutes and the transport report watchdog is one minute. A failed phase exits nonzero.
The recovery drill adds spectator authority, readiness, detach replay, and Direct reconnect histories:
Against the same pinned server, Godot 4.3 Linux arm64 passed four spectator cases: v2/v3 observers with JSON/raw MessagePack and a v3 reconnecting player. Observers receive authority release/transfer, current cached readiness and peer return notifications while retaining the spectator role. After a second player disconnect, a viewer leaves. The viewer receives one terminal reply with room identity and reason, clears its session and retained identity, and connected players remove it from their roster. The returning player receives one decoded detach event in replay with the same identity, reason and terminal roster. Replay emits no live detach signal. A fresh spectator join receives the restored roster and gameplay from the returning player. The pinned server pre-issues reconnect credentials only on v3; these cases do not claim v2 directed reconnect coverage.
Two further reconnect cycles overlap a Direct host endpoint update with peer reconnect. Both clients converge on the new endpoint and shared generation. Each fresh plan follows the reconnect baseline, the current credential rotates, and relay gameplay resumes. These are overlapping socket requests; they do not force an internal publication race. The roster recovery drill checks incumbent metadata on the newer server.
The recovery drill uses the same optional base URL, eight-second waits and
two-minute watchdog. Success prints LIVE_SERVER_ACCEPTANCE_PASS; any failed
phase exits nonzero. It passed against the pinned server above. Ordinary
voluntary detach and replay are covered. The publication drill below covers correlated recovery results and
authoritative removal for #356. The fault and drain drill below covers both local
completion panic points and terminal-before-close ordering.
The controlled publication and server delivery-pressure drills below cover
concurrent publication and queue policies. These drills are opt-in and
outside all. The mesh drill checks native and
browser WebRTC packet boundaries separately.
The pressure drill adds two-sender fan-in and selected-host departure:
It passed on Godot 4.3 Linux arm64 against server 182cb234. Four v2/v3
JSON/raw MessagePack cases queue 80 real socket frames at an incumbent
spectator while a second spectator joins. Dispatch drains one frame per poll.
Checks cover exact sender counts, per-sender order, one membership delta,
baseline-before-data, cached rosters and fresh gameplay. v2 carries the full
roster in membership deltas; v3 carries compact deltas with an empty roster
array (src/websocket/sending.rs at the pinned server commit). This measures client
receive backlog. The server delivery-pressure drill below covers queue
saturation and latest/volatile delivery.
A three-player Direct case removes the selected host. Both survivors receive a fresh shared generation and the successor's endpoint. Removing the remaining endpoints produces a shared Relay reset with no host, endpoint, peers or ICE. Plan callbacks see the reset and relay gameplay succeeds. Controlled publication is covered by the drill below. The publication drill below checks game-owned Direct socket replacement and teardown.
For #356, apply the local-only
spectator_detach_fault.patch
to an isolated checkout of server
182cb234c7aa601be3c2aad881948b3c5b853aaa. Build with Rust 1.94.0. The patch
inserts assertions inside the existing caught local-completion transaction.
Use it only for acceptance. It adds no fault switch to the shipped addon.
# From the isolated server checkout, with the config above:
git apply /path/to/signal-fish-client-godot/tests/fixtures/spectator_detach_fault.patch
cargo build --locked --bin signal-fish-server
SF_GODOT_DETACH_FAULT=after_remove SIGNAL_FISH__SERVER__DRAIN_GRACE_SECS=2 \
target/debug/signal-fish-server
# In another terminal, from the Godot client checkout:
godot --headless --path . --script tests/smoke/run_live_recovery_acceptance.gd
Repeat with SF_GODOT_DETACH_FAULT=after_local. Each mode passed all four
spectator recovery cases and the two Direct reconnect cycles. Server logs
confirmed the chosen fault and local recovery. The existing recovery oracle
checks exactly one terminal reply, live roster removal and replay detach,
room identity, reason, retained identity cleanup and fresh spectator gameplay.
These injections exercise the upstream repair; a request-triggered panic is
unproven. The controlled drill below covers registration retry/exhaustion.
The publication drill below covers authoritative removal.
The same isolated server can then run the drain case:
godot --headless --path . --script tests/smoke/run_live_pressure_acceptance.gd \
-- ws://127.0.0.1:3536 drain
When it prints LIVE_DRAIN_READY, send SIGTERM to that server process within
eight seconds. The drill has already queued the spectator terminal reply at
the engine peer. Both fault modes passed the exact signal order
SpectatorLeft, GoingAway, disconnected with close code 4000 and cleared
room/identity state. The server uses a two-second grace for this drill.
The publication drill below covers recovered correlated payload decoding;
SDK room-operation admission remains unsupported. These drills are opt-in.
Active gameplay shutdown¶
Use this POSIX-only local control for #482, #403 and #355. It starts and stops its own server processes. It runs native and Web seats with 4 KiB relay bodies and exact mesh packets with 4 KiB bodies in all three transfer modes.
Build server commit ee20b51ee605b09cdb448615f7500f6877d8c6e1 or a later
source containing the shutdown close repair.
Keep the build log and source state with the executable hash. Native runs need
Godot and the native WebRTC extension. Web runs need matching Godot Web export
templates and Playwright Chromium. The default all runtime needs both sets.
Use a local server config with WebRTC enabled and empty ICE servers, as in
docs/mesh-guide.md.
python scripts/check-active-shutdown.py \
--server-binary /path/to/signal-fish-server \
--server-config /path/to/local/config.json \
--server-revision ee20b51ee605b09cdb448615f7500f6877d8c6e1 \
--native-extension /path/to/webrtc_native \
--godot /path/to/godot --chromium /path/to/chromium \
--output /tmp/active-gameplay-shutdown
The output directory must be new. --runtime native|web and
--mode graceful|reset|stall select a control. The driver copies the config,
selects a local port and sets a three-second server drain. Other config values
remain supplied by the caller. The declared server revision is recorded beside
the executable hash; the driver cannot prove how an executable was built.
The default relay rate is four packets per frame per seat.
--relay-packets-per-frame 1 selects a slower comparison. Keep its result
separate from the original offered rate.
For a native engine built for another CPU, add --runner /path/to/emulator
and repeat --runner-arg=VALUE for each argument. For QEMU x86-64 on ARM64,
use --runner /usr/bin/qemu-x86_64 --runner-arg=-L
--runner-arg=/usr/x86_64-linux-gnu. This requires --runtime native and
--import-godot /path/to/native-editor. Only gameplay runs through the
emulator. The importer runs directly. Use an extension bundle containing
libraries for both CPUs. Provenance records the emulator hash and arguments
beside the engine, extension and importer hashes. Keep emulation results
separate from physical hardware results.
Each control first exchanges 32 exact mesh packets per mode and direction. The RPC peer caches an unreliable packet before a later reliable arrival. The drill checks both packets' bytes and modes, then caches another packet for retirement. Relay sends stay active through graceful drain.
graceful: SIGTERM must yieldGoingAway, then onedisconnected(4000, "server_shutdown")per client.reset: SIGKILL must yield an abnormal close or a send-side transport failure. Native and Web raw close values can differ.stall: SIGSTOP holds the server while each client callsclose(). The two-second close watchdog must emitheartbeat close timeout.
Every control checks that the mesh is empty, the cached receiver cannot return or send data, and terminal signals stay single. Graceful and reset controls check automatic retry eligibility, then disable retries for a controlled fresh login. User close must leave retries stopped. After server restart, a new plan and receiver must exchange every mesh packet again. The old receiver stays empty.
Check every send result. A Web send can return ERR_UNCONFIGURED while a close
frame is pending. The drill records that refusal and stops application sends
for that seat. Graceful shutdown still requires the exact semantic close.
Relay receives require exact bytes, sender and sequence for the delivered
prefix. Shutdown may discard the remaining relay tail.
Each case retains godot.log, server/server.log, assessment.json, import
logs and effective config. Web cases also retain export logs. Root provenance
records source and executable hashes. Failed attempts remain in their original
output directories. These are local acceptance controls; the
remote Web lane covers official Web 4.3,
4.4.1 and 4.7.2, and the
remote native lane covers official
4.4.1 and 4.7.2 plus the pinned repaired 4.3 backport. Other engine versions
and extension builds need separate qualification.
On Linux ARM64, one six-case cohort passes with official Godot 4.7.2, Chromium
156.0.8078.4 and server ee20b51e. Native seats use the opt-in repaired
extension described in docs/mesh-guide.md. The server executable SHA-256 is
4f9c94a766864be0a151263b43230518e62910c4dfe9b7b8d4defbcde82fbdd3.
The local reports are under /tmp/session452-final-matrix.
Later cold-editor imports abort before gameplay (#447). Those driver attempts
remain failed. Direct runtime boots of the exact prepared projects pass the
native graceful and nonresponding controls. The import failure remains open.
A Godot 4.3 native control with the earlier extension preserves shutdown
metadata and fresh mesh gameplay, but fails on a WebSocket receive-buffer
overflow and relay sequence gap (#485). Its reset and nonresponding controls
pass. The original failure remains under /tmp/session452-native43-original.
A separate one-packet-per-frame control passes every check. The slower pass
leaves the original reliable receive failure open. These results establish
the semantic closes on those local paths. A delayed browser control below
exposes a separate send/close race. Other engine coverage and extension
builds remain open in #482, #403 and #355.
Remote-path qualification (Web)¶
The Gameplay Shutdown Remote lane runs the three Web controls through a
credential-free Cloudflare quick tunnel. The server keeps binding locally;
both seats connect over wss:// through the Cloudflare edge, so every relay
message, the GoingAway advisory and the semantic close cross a real remote
path. The driver takes --service-port plus --service-public-origin for
this; the origin cannot be combined with the delayed local proxy. Four
consecutive green runs of both jobs - the post-merge main run
41905622779
plus three branch dispatches of the same lane (41898423659, 41900140061, 41904306119) - pinned the remote shutdown signature, so the lane carries
a weekly scheduled run (Mondays 13:00 UTC, staggered after the
WebSocket Remote Path watch) with the same pinned inputs. The weekly Web
run covers official Godot 4.7.2, 4.4.1 and 4.3; the native run covers 4.7.2
and 4.4.1. Each job takes its own dispatch list in web_engine_version and
native_engine_version (space or comma separated) to qualify another
target or re-run one engine. Dispatch runs keep on-demand qualification
available; GitHub disables schedule triggers
after ~60 days without repo activity. A red run triages from the run page:
a drill failure reads the run summary alone (the known #447 import-abort
attribution first; the native loop already re-dispatches import-unstable
attempts and retains each), while a tunnel-provisioning failure points at
the failed tunnel step (run
41897404439
is the transient record).
Official 4.4.1 Web qualified in the two-engine dispatch
41984709949
(Chromium 156.0.8078.4, 32 KiB application-ACK window). All three controls
passed: graceful delivered GoingAway then exactly one 4000/server_shutdown
close per seat, reset closed 1006, and stall ended with the heartbeat close
timeout. Both gameplay generations completed in every mode, each case counted
two service starts and one shutdown request, and provenance recorded the
tunnel origin.
Official 4.7.2 repeated green in the same run.
Official 4.3 Web qualified in dispatch
41991293515
(Chromium 156.0.8078.4, 32 KiB application-ACK window). All three controls
passed: graceful delivered GoingAway then exactly one 4000/server_shutdown
close per seat, reset closed 1006, and stall ended with the heartbeat close
timeout. Both gameplay generations completed in every mode, and each case
counted two service starts and one shutdown request. The 4.7.2 and 4.4.1 Web
jobs repeated green in the same run. The native job keeps its 4.7.2 and
4.4.1 pair: official 4.3 native still hits the #485 receive loss at the
original uncredited rate.
Remote-path qualification (native)¶
The lane's native job runs the three native controls through the same kind
of quick tunnel. One matrix entry builds the pinned repaired 4.3 backport
from source (below); the rest are official releases. The official editor
imports each case project and runs it headless; the pinned official WebRTC
GDExtension backs its peer
connections. The lane pins release 1.2.2-stable by zip SHA-256; that tag
is commit 62135fc034cbdc5aee177136e526e3ead3cc563a, the source revision
other qualifications build from. The official release binary itself is new
qualification ground: earlier native #482 cohorts ran opt-in repaired
extension builds. The gate compares each case's staged extension library
hash against the release hash after the drill, so a substituted library is
detected and fails the qualification; the exact gameplay checks are what
qualify the peer behavior.
The native seats use the original uncredited workload (four 4096-byte
relay bodies per frame), the control the local native qualifications used;
a relay loss over the edge is new #488-family data, not a close-metadata
failure. The per-case cold-editor import remains the known-open #447 abort
hazard; the gate attributes an import abort so a red run is readable as
engine-import data before it is read as remote-path data. The lane also
re-dispatches import-unstable attempts (retaining each) and gives every
editor invocation a fresh HOME with --fresh-home: the captured aborts
all land right after the editor loads its layout, which reads shared
state under the caller's HOME.
Official 4.4.1 native qualified in the same dispatch
(41984709949)
with the pinned release extension (library SHA-256
d99224f5411c5083f1d0932f2ed30b0b68f5f697be025671ff0083015f06a033).
All three controls passed at the original uncredited rate: graceful
delivered GoingAway then exactly one 4000/server_shutdown close per
seat, reset reported the raw abnormal close -1, and stall ended with the
heartbeat close timeout. Both gameplay generations completed in every
mode, and each case counted two service starts and one shutdown request.
The native list also carries the pinned repaired 4.3 backport
(77dcf97d82cbfe4e4615475fa52ca03da645dbd8,
scripts/backport-websocket-43.py). The job builds a template_debug
x86-64 engine with GDScript, WebSocket, WebRTC and mbedtls, imports each
case with the official 4.3 editor, and runs the same three controls at the
original uncredited rate (four 4096-byte bodies per frame, no credit). This
is the 4.3 engine that holds the #485 receive boundary; official 4.3 does
not. Dispatch
42076926873
passed all three: graceful delivered GoingAway then exactly one
4000/server_shutdown close per seat, reset reported -1, and stall ended
with the heartbeat timeout. Both generations completed in every mode, and
each case counted two starts and one shutdown request. The gate compares
the run's engine hash with the freshly built binary and checks the
custom-build version string and the extension library; this run used
SHA-256
8aebdbce741747e08ba00ec694764e38d9d311ee94e5193d582c447197d831f3. The
first dispatch
(42074629297) passed every control but failed its gate on the 4.3 version
string (4.3-stable (custom_build)); that attempt stays retained. A
dispatch sets native_repaired_4_3 to skip the build. Upstream adoption
stays open in #536.
Native WebSocket receive boundary¶
Issue #485 is reproducible with WebSocketPeer and the local smoke server.
This native control needs no Signal Fish server or WebRTC extension:
It writes 64 exact 4096-byte messages, holds client polling for 100 ms, then drains. A later sentinel exposes an interior gap after a dropped burst tail. Every write must succeed. Every received packet must match its bytes, string mode and sequence. The final count must be 65. Engine errors remain failures. The watchdog covers the five-second connection window, each batch hold/read window and the final sentinel read. It is opt-in because the original Godot 4.3 failure must remain visible.
Use -- --credit=32768 to split the same finite work into batches bounded by
local consumer body bytes. Use -- --capacity=524288 for a larger byte buffer,
or add --max-queued-packets=2 to isolate the packet-count boundary.
This native byte credit bounds bytes only; it does not bound packet count.
Credit uses direct local consumer observations; it adds no network ACK protocol.
The script prints each poll, packet and final verdict. Keep logs, commands,
source hashes and engine hashes for every attempt.
On Linux ARM64, the new controls retain these results:
| Engine | Default buffer | 512 KiB buffer | 512 KiB, two packets | 32 KiB body credit |
|---|---|---|---|---|
| 4.3 | Fails: 16/65 | Pass: 65/65 | Fails: 3/65 | Pass: 65/65 |
| 4.4.1 | Pass: 65/65 | Pass: 65/65 | Pass: 65/65 | Pass: 65/65 |
| 4.7.2 | Pass: 65/65 | Pass: 65/65 | Pass: 65/65 | Pass: 65/65 |
| 4.3 backport | Pass: 65/65 | Pass: 65/65 | Pass: 65/65 | Pass: 65/65 |
The default byte setting is 65535 in all three engines. Matching the newer engines to 2048 queued packets also passes. The changed default packet count alone does not explain the repair. Godot's unbuffered receive repair limits socket reads to remaining queue space. Godot 4.4.1 includes this repair and later fragmented-message fixes. An SDK drain cannot recover bytes already dropped inside a Godot 4.3 engine poll. A larger finite buffer only moves that boundary. The retained 4.3 failure still needs an engine upgrade or qualified backport for a receive-boundary repair.
Add -- --fragment-bytes=127 to split each message into WebSocket TEXT and
CONTINUATION frames. Zero preserves single-frame messages. Message bytes,
credit, capacity and the later sentinel stay unchanged. TCP may coalesce the
writes. Godot 4.4-beta1 passes the original control but truncates fragmented
messages: a 512 KiB run receives 65/65 messages with 127-byte bodies instead of
4096 bytes. The exact-byte check fails from packet zero. Godot 4.4.1 and 4.7.2
pass the same fragmented controls. Add --message-type=binary for bodies
containing all byte values, or --message-type=alternating to alternate text
and binary messages. Both bytes and packet type must match the offered order.
The default text workload stays unchanged. A 4.3 backport also needs the
message-completion callback repair,
the
fragmented-message assembly repair,
and the
closing-buffer lifetime repair.
The assembly change alone still commits individual fragments. Clearing the
receive buffers before the close handshake blocks its socket reads.
The beta is a historical negative control; it adds no support promise.
Add --fragment-control=ping or --fragment-control=pong with a positive
fragment size to place a nonempty control frame inside each fragmented
message (#491). Control bytes must stay out of the delivered text. Official Godot
4.4.1 and 4.7.2 fail this exact-byte control even when all 65 packets arrive.
The buffered Godot 4.3 capacity control preserves the message bytes.
Apply the experimental native backport to a clean Godot 4.3-stable source checkout with:
python scripts/backport-websocket-43.py --source /path/to/godot \
--report /tmp/websocket-backport.json
The script pins the source revision and all four upstream commits. It records
patch and source hashes, rejects conflicting edits, and supports --check.
Build the engine separately and retain its command, log and binary hash.
The experimental patch also keeps interleaved Ping/Pong bytes out of the
fragmented message and clears pending assembly on connection replacement.
This patch does not change the Web callback receive boundary.
Issue #492 covers a separate direct-peer reuse failure. Run
godot --headless --path . --script tests/smoke/run_websocket_fragment_reuse.gd
to replace a closing peer after an unfinished fragment. The fresh text must
be exactly five bytes. Official 4.4.¼.7.2 retain stale assembly length and
return an empty packet with a buffer error. The SDK creates a fresh engine peer
for each session; this direct-reuse result does not establish an SDK reconnect
defect. Pass script options after Godot's -- separator:
godot --headless --path . \
--script tests/smoke/run_websocket_fragment_reuse.gd -- \
--partial-type=binary --fresh-type=binary
Choose the partial and fresh message types independently. The fresh binary body includes zero and non-UTF-8 bytes.
Separate experimental patches target Godot 4.4.1-stable
49a5bc7b616bd04689a2c89e89bda41f50241464, 4.5.2-stable
6ce3de25aa58466e14ef354703ba8d9791a417da, 4.6.3-stable
35e80b3a8822a9df9be390814b62f44c0a9c69e8, 4.7.2-stable
ed1daf0bf001b61586d9930840f2f1394092c079 and master
95c0fa5fe8a019adc29d074c16b64469b04571f7:
python scripts/repair-websocket-stable.py --source /path/to/godot-stable \
--report /tmp/websocket-assembly.json
Each patch repairs receive progress, frame assembly and replacement cleanup. The applicator
checks the exact revision, selects its patch, rejects conflicting changes and
records source and patch hashes. --check leaves source unchanged. Build and
qualify the patched engine before use. The master pin carries the same repair
submitted upstream
(godotengine/godot#125066);
its four defects and evidence map to #491, #492, #496 and #498. Official
4.4.1 and 4.7.2 still fail
the mixed binary interleaved controls and every partial/fresh text/binary reuse pair. Further engine
builds and remote gameplay remain open.
A weekly scheduled run (Mondays 09:00 UTC, WebSocket Master Pin) keeps the
master pin aligned with upstream. It compares the pin with current
godotengine/godot master and prints the state of godotengine/godot#125066
for review tracking. When master moves, it mirrors the patched files at the
new revision and checks that the bundled patch still applies. A moved master
with an applying patch passes. A red run names the outcome: the receive path
drifted (rebase the patch, refresh the submission) or the repair was adopted
in full (the patch reverse-applies; requalify the adopted revision, then
retire the pin).
The experimental 4.5.2 Linux ARM64 template passes all seven standalone reproductions below. Official 4.5.2 fails the four defect cases and passes the three comparison controls. These checks cover fragmented Ping/Pong bytes, CLOSING peer reuse, full-buffer completion and split Unicode close reasons (#491, #492, #496, #498). The build also passes all 181 paced and large-boundary cases, normal WebSocket smoke and all three delayed active shutdown/recovery modes. Other build targets and remote networks remain open.
The experimental 4.6.3 Linux ARM64 template also passes all seven standalone
commands. A matched unpatched build from the same source revision and build
options fails all four defect commands and passes the three comparison controls.
The checksum-verified 4.6.3 release asset fails 108 of the 181 receive cases.
That asset reports commit 7d41c59c4; the source builds use 35e80b3a8.
Keep these baselines separate. The repaired build passes all 181 receive cases,
normal WebSocket smoke and all three isolated active shutdown/recovery modes.
An initial concurrent graceful attempt hit outbound send backpressure; the
initial stall setup aborted during editor import (exit -6; #447). Retain both
failed attempts. The isolated passes do not explain either failure.
The same 4.6.3 repair also passes all 181 receive cases, all seven standalone commands and SDK WebSocket smoke on Linux x86-64 under QEMU 8.2.2 on ARM64. The matched unpatched x86-64 build uses the same source revision and compiler options. It reproduces packet corruption, stale reuse state, a full-buffer stall and code 1007 for a valid Unicode close. All three comparison controls pass. These are local emulation results. Physical x86-64 hardware and remote networks remain open.
The 4.7.2 repair also passes all seven standalone commands on Linux x86-64
under QEMU 8.2.2 on ARM64. Matched builds use revision
ed1daf0bf001b61586d9930840f2f1394092c079 and identical compiler options.
The unpatched build reproduces all four defects and passes all three comparison
controls. This covers fragmented Ping/Pong bytes (#491), peer reuse (#492),
full-buffer completion (#496) and Unicode close reasons (#498).
The repair passes all 181 receive cases, SDK WebSocket smoke and all three
active shutdown/recovery modes (#482). The active drill uses 50 ms proxy delay,
127-byte chunks and a 100 ms receive pause with the original deadlines.
Each mode preserves exact mesh traffic across both connection generations
and an exact delivered relay prefix. Physical x86-64 hardware, remote networks
and upstream adoption remain open.
For 4.6.3 and 4.7.2 template builds, add disable_path_overrides=no to the SCons command
to enable --path and --script for these fixtures. Use the matching official
editor to import the test project before running a template build. The active
shutdown drill also needs module_multiplayer_enabled=yes alongside GDScript,
WebSocket and WebRTC.
To check the four bundled stable patches against local pinned sources, run:
PYTHONPATH=tests \
SF_WEBSOCKET_GODOT_44_SOURCE=/path/to/godot-4.4.1 \
SF_WEBSOCKET_GODOT_45_SOURCE=/path/to/godot-4.5.2 \
SF_WEBSOCKET_GODOT_46_SOURCE=/path/to/godot-4.6.3 \
SF_WEBSOCKET_GODOT_47_SOURCE=/path/to/godot-4.7.2 \
python -m unittest test_websocket_browser_observation.BackportApplicatorTests.test_bundled_patches_apply_to_pinned_sources
This opt-in test uses temporary local clones to check, apply and repeat each real patch. The source checkouts stay unchanged. Normal CI skips this test.
The locally built 4.4.1 native template passes 31 controls with no engine errors: text, binary and alternating messages across no-control/Ping/Pong, default byte limits, two queued packets and local 32 KiB credit, plus all four partial/fresh message-type pairs. Every burst delivers 65 exact packets with matching types. Reuse delivers exactly five fresh bytes. The same build passes the standard WebSocket smoke with client/server close handshakes. Its active SDK drill passes semantic shutdown, reset and nonresponding controls with the original uncredited four-per-frame sends, 50 ms TCP delay, 127-byte proxy writes and a 100 ms polling hold. Exact relay prefixes, cached mesh retirement and fresh gameplay receipts remain required. These results qualify the pinned Linux ARM64 build and local workload.
The locally built 4.7.2 native template passes the same 31 controls, close smoke and all three active SDK shutdown modes with the same uncredited sends, TCP delay, proxy writes and polling hold. Every burst delivers 65 exact 4096-byte packets; each reuse pair delivers five fresh bytes with the correct type. A matched official 4.7.2 run passes nine no-control burst cases and fails all 18 interleaved-control cases and four reuse pairs. Its active shutdown controls pass. These results qualify the tested Linux ARM64 build and local workload. Remote paths, other targets and upstream adoption remain open.
Use --import-godot /path/to/official-godot-4.3 in the active-shutdown driver
to import with an editor and run a native template build via --godot.
Both engine binaries are hashed in the retained results.
The locally built 4.3 template passes all three active-shutdown modes with the original uncredited four-per-frame sends, 50 ms TCP delay, 127-byte proxy writes and a 100 ms polling hold. Receive capacity stays at 65535 bytes. Exact relay prefixes, semantic/reset/nonresponding close outcomes, cached mesh retirement and fresh exact gameplay remain required. This qualifies the tested native build and local workload; upstream and remote qualification remain open. A new matched official 4.3 shutdown control also passes; that passing comparison does not explain the retained overflow history. The finite burst controls keep the native receive-boundary failure explicit.
Full fragmented-message receive limit¶
A fragmented message can fill the native receive buffer before its final empty continuation arrives (#496). Official Godot 4.4.¼.7.2 and the earlier experimental native patches stall with an open connection and no complete packet. Run:
The fixture fills the 4095-byte buffer with an unfinished binary message,
then sends an empty final continuation and a separate two-byte binary message.
Both messages must arrive with exact bytes and types. Use
-- --message-type=text --fragment-control=ping for text and an exact Pong
response. --fragment-control=pong checks incoming Pong separation.
--capacity=65535 checks the default byte limit.
--oversize=true adds one continuation byte: the peer must close with code
1009 and return no packet. Engine buffer errors fail every control.
The updated experimental patches permit one byte per socket read when an unfinished message fills the payload buffer and no complete packet is queued. This lets empty continuations and controls progress without reading into the next message. The bundled wslay change tracks cumulative message length in unbuffered mode, so an extra continuation byte still exceeds the configured limit. The 4.3 patch also adopts the pinned 4.4.1 close handling for a stopped parser, so a rejected message reaches a terminal close state. Apply the current patch to a clean pinned source checkout and rebuild. Earlier patched binaries do not contain this repair.
The burst fixture also accepts --body-bytes=4095 --capacity=4095 and
--empty-fragments=true --fragment-bytes=127. Empty fragments surround each
message; the default 4096-byte workload stays available.
The reuse fixture accepts --replacement=closing|aborted|handshake and
--queued-before-partial=true. It checks that a complete unread packet and
unfinished fragment are retired before the fresh connection's five-byte
message.
The rebuilt Linux ARM64 templates for 4.3, 4.4.1 and 4.7.2 each pass 84 receive/reuse controls, WebSocket close smoke and all three active SDK shutdown modes. These local results extend #485, #491 and #492. Remote networks and upstream adoption remain open.
Paced native WebSocket reads and close reasons¶
Use the opt-in native runner to split frame headers and bodies across local TCP writes. It covers full-buffer completion (#496), Ping/Pong separation (#491), peer replacement (#492), and valid Unicode close reasons (#498):
python scripts/check-websocket-native-receive.py \
--godot /path/to/repaired-template --importer /path/to/godot-editor \
--output /tmp/native-receive-attempt
Use a new output directory for every attempt. Reports retain commands, logs, fixture hashes, executable hashes, exact result records and timeout failures. On native Linux, add a syscall read control to keep TCP coalescing from joining all the paced writes into one receive:
python scripts/check-websocket-native-receive.py \
--godot /path/to/repaired-template --importer /path/to/godot-editor \
--recv-cap-bytes=3 --output /tmp/native-receive-cap3
This option requires cc and a dynamically linked Linux engine. The runner
builds a local LD_PRELOAD helper that limits each libc recv call on both
fixture endpoints. The editor import runs without the helper. Choose an output
path without whitespace or colons. The option supports native execution;
QEMU runs use the write controls above.
Each attempt keeps the helper source, compiled library, build command, hashes, child environment and per-case read counters. A case requires observed limited reads, a maximum read size within the chosen cap and all existing packet/close checks. Missing observations fail the attempt. Existing workloads and deadlines stay unchanged. These controls qualify local read boundaries; remote paths need separate runs.
Use --case=read-control-none, --case=read-control-ping and
--case=read-control-pong for the roomy-buffer comparison. These three cases
keep each fragmented 4095-byte message below the receive capacity. They isolate
control-frame assembly from the full-buffer completion cases.
On Linux ARM64, official 4.7.2 fails both roomy Ping/Pong controls with a three-byte read cap: a 4095-byte message arrives with 4114 bytes. The no-control case passes. With the same cap, a complete-frame Unicode close write becomes code 1007 in both directions; ASCII and empty reasons pass. The repaired 4.3, 4.4.1 and 4.7.2 builds pass the roomy controls. These results qualify local syscall boundaries.
To test a cross-compiled engine with QEMU, use an explicit runner:
python scripts/check-websocket-native-receive.py \
--godot /path/to/x86-64-template --importer /path/to/native-editor \
--runner /usr/bin/qemu-x86_64 \
--runner-arg=-L --runner-arg=/usr/x86_64-linux-gnu \
--output /tmp/native-receive-x86-attempt
The importer runs on the host. Reports keep the target engine hash, runner hash and runner arguments separately. Keep the cross compiler and sysroot versions with the build record. Emulation results apply to local emulation; physical hardware and remote networks need separate qualification.
Use --suite=assembly, --suite=close or --suite=boundaries to select a
group; --case=NAME selects individual cases. The full matrix has 184 cases.
The boundaries group adds 55 cases for large gameplay and control bodies:
- 131071-byte messages use 64-bit frame lengths. Text, binary and Unicode bodies require exact full-buffer completion or code 1009 for an extra byte.
- Unicode characters span continuation boundaries with up to three tail bytes. Interleaved Ping/Pong bodies contain all byte values from 0 through 124. Ping requires an exact 125-byte Pong.
- Maximum-length close reasons contain 123 UTF-8 bytes. ASCII and mixed two-/three-/four-byte text are checked in both directions with an empty packet queue or an unfinished full-buffer message. Client-initiated close frames must preserve the exact reason bytes as well as the code.
The receive-limit fixture accepts --capacity=131071,
--message-type=unicode, --tail-bytes=1|2|3 and --control-bytes=125.
The close fixture accepts --reason=max-ascii|max-mixed and
--capacity=127|131071. These optional controls retain the original cases.
The three receive/reuse fixtures accept --wire-chunk-bytes=1 and
--wire-delay-ms=10. The sender queues frame bytes, writes at most one chunk
per server poll, honors partial writes and retires queued output when the
connection changes. Delay alone selects 65536-byte chunks. The default sender
writes each complete frame immediately. Reports include successful writes,
bytes, split headers, pending bytes, retired bytes and write errors.
TCP can coalesce writes; this is local scheduling qualification.
The new close fixture checks exact code 3400 and valid empty, ASCII, two-, three-, four-byte and mixed UTF-8 reasons. For a focused reproduction:
godot --headless --path . \
--script tests/smoke/run_websocket_close_reason.gd -- \
--wire-chunk-bytes=3 --wire-delay-ms=10 --reason=two --origin=server
Use --origin=client for a client close with a paced server echo and
--full-fragment=true to close with an unfinished 31-byte message filling
the receive buffer. Both directions require the exact close reason, terminal
state, no application packet and the matching client close code.
Bundled wslay skips the first two close-payload bytes during every receive chunk's UTF-8 validation. Those status bytes occur once per close frame. A valid Unicode reason split across reads can therefore produce code 1007. The current experimental 4.¾.4.¼.7.2 patches skip only the status bytes that remain at the accumulated payload offset. Rebuild from the current patches; earlier repaired binaries still contain this close-reason defect. Upstream adoption, other targets and remote networks remain open.
Standalone upstream reproductions¶
Create a small project for engine bug reports:
Extract the ZIP and follow README.txt. The project needs only Godot and
uses a local raw TCP server. It includes exact-byte Ping/Pong assembly,
peer replacement, full-buffer completion and split Unicode close controls,
plus passing comparison commands. SHA256SUMS pins every input. The exporter
refuses an existing archive and accepts no runtime options.
Godot closed the upstream reports because it does not accept AI-authored
issues. This repository keeps the maintained record in
.llm/research/godot-websocket-receive-defects.md:
| Failure | Local record |
|---|---|
| Ping/Pong bytes enter fragmented messages (#491) | defect 1 |
| Closing peer replacement retains old state (#492) | defect 2 |
| Full-buffer empty continuation stalls (#496) | defect 3 |
| Split Unicode close reasons fail (#498) | defect 4 |
The extracted project's seven commands were checked on official and repaired
4.¾.4.¼.7.2 Linux ARM64 engines. All three repaired builds pass every
command. Official 4.4.¼.7.2 fail the four defect cases and pass the three
comparison controls. Buffered 4.3 passes assembly/reuse; its full-buffer case
drops the following packet. Current Godot master 7be59b46 retains the four
source paths; no master runtime result is claimed. Upstream adoption, other
targets and remote networks remain open.
The driver can also run those seven commands itself and retain one report:
python scripts/check-websocket-native-receive.py \
--godot /path/to/engine --importer /path/to/editor \
--run-repro --output /tmp/standalone-run
standalone-report.json pins engine and importer hashes and records optional
emulator paths and arguments. It records each command's exit and result
records. Each command's log sits beside the report as <name>.log. An engine
that crashes a command leaves no result, or an empty record when its result
line is cut short. Either way the gap stays visible. The driver exits nonzero
when the fixture import fails, including an import that exits 0 after logging
an ERROR: line. It writes the report before it exits. Only the import sets
the exit status: a failing reproduction command still exits 0, because
official engines fail the defect commands by design. Callers compare the
report with the expected matrix.
x86-64 CI qualification¶
The opt-in WebSocket Native x64 workflow builds a pinned engine revision on
an x86-64 GitHub runner and qualifies it there. Dispatch it with a pinned
stable or master revision to build the repaired engine, run the full paced receive
suite with a three-byte read cap, and run the seven standalone commands. Set
the matching official release label for the fixture import; a master dispatch
uses the nearest official release label because no release matches a master
SHA. A build_target
input selects linuxbsd (default) or windows; Windows builds
cross-compile with mingw-w64, run under Wine, and use the uncapped suite. A
repair_kind input selects the repair applicator. stable (default) patches
the pinned 4.4.1, 4.5.2, 4.6.3, 4.7.2 and master revisions with
repair-websocket-stable.py. 4.3-backport patches 4.3-stable
77dcf97d82cbfe4e4615475fa52ca03da645dbd8 with backport-websocket-43.py;
dispatch that run with the matching revision and the 4.3-stable label. The
4.3 remote leg pairs 4.3-backport with run_baseline and remote_path on.
The
4.3 build drops the disable_path_overrides SCons flag, which 4.3's
SConstruct predates. The lane rejects a stable source whose label names a
different major.minor family; a master pin keeps the nearest official
release label. A
matched unpatched baseline job runs the same seven commands, each keeping
exactly one observed result record. The defect commands fail on official
4.4.1, 4.5.2, 4.6.3 and 4.7.2 baselines. Official 4.3 passes the
control-bytes and reuse repros unpatched, so its baseline fails the
full-buffer and close-unicode commands. The three comparison controls must
pass. Both jobs upload build logs, patch reports, engine
hashes and per-command evidence. A remote_path input adds the
remote-network leg: each variant fronts a Cloudflare quick tunnel and drives
the ten pinned receive cases over wss:// (see
Remote path qualification). The build enables
the mbedtls module so the engine can connect over TLS at all.
The first run qualified the pinned 4.7.2-stable repair
ed1daf0bf001b61586d9930840f2f1394092c079 on a virtualized x86-64 host that
executes the engine natively (run
41251544967).
The repaired engine passes all 184 paced receive cases with a three-byte read
cap; every case observed capped reads. All seven standalone commands pass
(#491, #492, #496, #498). Repaired binary SHA-256:
c422f1a4b40b5b5f52cb6c87cc7984d661bb4925d625c7b390d7c161bf65a27d. The
matched unpatched build from the same source and flags reproduces all four
defect commands with one observed result record each. The three comparison
controls pass. Unpatched binary SHA-256:
e3ffcc28cfa930b4688367ebbabef4037c7255a4ae93d689599c784eaa4c0c6e. SCons
4.11.1 builds both engines with the options listed in the workflow. These
results cover the #485 receive suite, paced writes, large boundaries, close
reasons, peer replacement and local syscall read caps on x86-64. Keep them
separate from QEMU emulation runs. Bare-metal hardware and upstream
adoption remain open, and the macOS leg stays with the weekly
WebSocket Native macOS watch. The 4.4.¼.5.2/4.6.3 Linux remote legs
and the Windows remote dispatch are recorded under
Remote path qualification.
The pinned 4.4.1-stable repair 49a5bc7b616bd04689a2c89e89bda41f50241464
passes the same checks on x86-64 (run
41256780905).
All 184 paced receive cases pass with capped reads and all seven standalone
commands pass (#491, #492, #496, #498). Repaired binary SHA-256:
3db399d851cd054152024c902107521eeb42155845c9b22f07747caa16fe093f. The
unpatched baseline reproduces the four defect commands and passes the three
controls. Unpatched binary SHA-256:
d01959e694c2c4c16855ab1faa4c8d4b4d15f01440d3a20c1c37bb7bf49d92a7.
The pinned 4.5.2-stable repair 6ce3de25aa58466e14ef354703ba8d9791a417da
passes the same checks on x86-64 (run
41256782273).
All 184 paced receive cases pass with capped reads and all seven standalone
commands pass (#491, #492, #496, #498). Repaired binary SHA-256:
03cbf9c4f87f5bab3ca22231992fc7abcb72ae1ca518d57d2a812cdab0eb86f7. The
unpatched baseline reproduces the four defect commands and passes the three
controls. Unpatched binary SHA-256:
0af7ced3589f5046f9b19d7d1c0fef5b49d2a98142da82758632a8ea7895d4c9.
The pinned 4.6.3-stable repair 35e80b3a8822a9df9be390814b62f44c0a9c69e8
passes the same checks on x86-64 (run
41256784292).
All 184 paced receive cases pass with capped reads and all seven standalone
commands pass (#491, #492, #496, #498). Repaired binary SHA-256:
7a1432456df8b78940200bf14f5ffc8c49e65cc12986df6814e4518701a59002. The
unpatched baseline reproduces the four defect commands and passes the three
controls. Unpatched binary SHA-256:
566db2f263e02fcd266a99535142f156133c151d761f9b905cbbae612eeda466.
The pinned 4.3-stable backport 77dcf97d82cbfe4e4615475fa52ca03da645dbd8
passes the same checks on x86-64 (run
41730274982).
All 184 paced receive cases pass with capped reads and all seven standalone
commands pass (#491, #492, #496, #498). Repaired binary SHA-256:
8aebdbce741747e08ba00ec694764e38d9d311ee94e5193d582c447197d831f3. The
unpatched 4.3 baseline passes the three controls, the control-bytes and
reuse commands, and fails the full-buffer and close-unicode commands,
matching the collected Linux ARM64 baseline results. Unpatched binary
SHA-256:
68c4bce9bca9c3947428069ab1d30ad5c2c2ebd6ee5c7acbef9749f293fc70df.
The windows build target qualifies the repair on Windows x86-64 under
Wine. The workflow installs current Wine from the WineHQ repository;
Ubuntu 24.04 Wine 9.0 cannot start Godot 4.7.x Windows binaries, official
or self-built. The pinned 4.7.2-stable repair
ed1daf0bf001b61586d9930840f2f1394092c079 passes all 184 receive cases
without read caps (run
41273388162).
All seven standalone commands pass. The matched unpatched baseline
reproduces the four defect commands and passes the three comparison
controls (#491, #492, #496, #498). Repaired binary SHA-256:
918124422ff1fa405042e8df31f91bfc49acb3ebe9a186a341a70f261634b3b0.
Unpatched binary SHA-256:
759cb4131ba251ee3e93593179a64f40e26d6acfc6d79661af1f9f02ad3e2874.
The pinned 4.5.2-stable repair 6ce3de25aa58466e14ef354703ba8d9791a417da
passes the same uncapped checks on Windows x86-64 under Wine (run
41278234161).
All 184 paced receive cases pass and all seven standalone commands pass
(#491, #492, #496, #498). The unpatched baseline reproduces the four defect
commands and passes the three controls. Repaired binary SHA-256:
4266d1d6ddae49e6967b3ae7073525a79f7f847ac5859346c3aedb42a13b5ef4.
Unpatched binary SHA-256:
daa3a46b4f1f146a4cd2fa96690bea3ba9e773713ce5600f8fbad3697b9decc4.
The pinned 4.6.3-stable repair 35e80b3a8822a9df9be390814b62f44c0a9c69e8
passes the same uncapped checks on Windows x86-64 under Wine (run
41278235547).
All 184 paced receive cases pass and all seven standalone commands pass
(#491, #492, #496, #498). The unpatched baseline reproduces the four defect
commands and passes the three controls. Repaired binary SHA-256:
5fc7e44f8cdd2dbaef133ba13fe61febbcc3d6d5cb2a1e91a7ca7882d64ea85c.
Unpatched binary SHA-256:
a00ba934fa2255a486a5a87b95682caba39db1524931a3da8d8c7fb10bd1ccc4.
The pinned 4.4.1-stable repair 49a5bc7b616bd04689a2c89e89bda41f50241464
passes the same uncapped checks on Windows x86-64 under Wine (run
41280052786).
All 184 paced receive cases pass and all seven standalone commands pass
(#491, #492, #496, #498). Repaired binary SHA-256:
90739b2e631da58757dc5fa723ca858fbe3c372a841269cb18b8992cc36305ca. The
unpatched baseline in run
41278232278
reproduces the four defect commands and passes the three controls. Unpatched
binary SHA-256:
f74caff33936d986323b2bff0620cb4d46144a268849d5a241995679ee89778a. An
earlier repaired build of the same revision in that run (SHA-256
93713f1b1bee33d9dd356c92c80cb1af04dfb0b6e7d3d6f7c4522f7270d9f6f8) missed
the suite's one-second finite-burst deadline once in
chunk-burst-alternating-pong: message 40 needed 1.14 s after message 39
while earlier messages averaged about 0.5 s, the deadline expired, and the
harness aborted the remaining bursts and offered only the sentinel. All 41
offered messages arrived with exact bytes and mode. The repeat run delivered
all 65 packets; the single deadline miss stays open pending recurrence
(#520).
A weekly scheduled run (Mondays 08:00 UTC) keeps that watch active: it builds
the repaired 4.4.1-stable engine for Windows x86-64 and runs the paced suite
plus the seven standalone commands without the baseline job. Dispatch runs
keep every input and both variants. Each burst case report carries a
delivery summary with the packet count, poll count, largest inter-delivery
gap and largest poll interval, parsed from the fixture log. When a batch
deadline expires, the burst result records a pending_wait entry with the
first missing sequence (or the delivery count when no gap was observed), the
elapsed wait and the deadline margin. The batch deadline is three seconds
(#520): the healthy Windows x86-64 runs never needed more than 563 ms
between consecutive deliveries, and the recorded one-off Wine pacing stall
needed 1.14 s, so the
former one-second window left under twice the headroom and let a benign
stall fail the watch. A deadline miss still names a stall above the window
or a lost message; compare both records with the batch deadline before
changing any deadline. The suite log prints a
delivery line for every passed burst case, and a failing case in either the
suite or the standalone log prints a diagnostics line with its result
record, so a red run triages from the CI log alone. The uploaded artifacts
keep the complete per-case evidence.
Windows-native CI qualification¶
The windows build target runs under Wine by default, which cannot show that
the repair holds on the Windows socket path itself. Dispatch the workflow
with build_target=windows and windows_native_runner=true: the Linux job
uploads the repaired engine, and a windows-latest job downloads it,
installs the official Windows editor through
.devcontainer/install-godot.py, and runs the uncapped paced suite plus the
seven standalone commands. The gate requires every case and reproduction.
Dispatch 42132635865
(Windows Server 2025, windows-2025-vs2026 image) built the pinned
4.7.2-stable repair ed1daf0bf001b61586d9930840f2f1394092c079 and ran the
same engine binary on both paths:
- Wine leg: 184/184 paced cases and 7/7 standalone commands passed.
- Native Windows leg: 184/184 paced cases and 7/7 standalone commands passed; both fixture imports exited 0 with no engine errors.
- Engine SHA-256
51115c67ff60d889c7f79f3b90c88234a96b0d4bab8f05927ad93516e25fde67; extracted official Windows editor executable SHA-256ab1824f85bfd8e0e4128182c000c4003a3e042245b2967848d089b2a04b22424.
The gate checks the downloaded engine against the uploaded checksum and the suite provenance, so both paths run the same binary. The native leg takes about six minutes after the build. It stays dispatch-only; the weekly Wine watch remains the recurring Wine check, and native Windows runs on dispatch. This qualifies the Windows socket path for the pinned build, not an MSVC rebuild or another engine family.
macOS CI qualification¶
The WebSocket Native macOS workflow qualifies the same pinned repairs on
Apple Silicon (Mondays 11:00 UTC). It builds the repaired engine on a
macos-15 runner with the minimal headless feature set (vulkan=no metal=no
opengl3=yes), runs the uncapped 184-case paced receive suite and the seven
standalone commands. The official macos.universal editor imports the
fixtures; its binary runs from inside the extracted Godot.app bundle
(the bare bundle binary will not exec on Apple Silicon). Dispatch runs
accept a pinned stable or master revision and can add the matched
unpatched baseline, which must reproduce the four defect commands
(#491, #492, #496, #498) and pass the three controls.
The pinned 4.4.1-stable repair 49a5bc7b616bd04689a2c89e89bda41f50241464
passes all seven standalone commands on macOS arm64 (repaired binary
SHA-256 5bfe14836ec8e33390c85cdd145c456819b7240d683f9f3f166f483420a91308).
The matched unpatched baseline reproduces the four defect commands and
passes the three controls on the same runner, so the platform gap sits in
the engine, not the fixtures. The uncapped paced suite holds a measured
signature across qualification runs (41510773288,
41516492113),
and the workflow asserts it:
- Twenty stable failures: every one-byte-paced
byte-reuse-*case misses the barrier pong (#492 family), and every one-byte-pacedboundary-close-1-max-*case never delivers a close code (#498 family). Short close reasons (empty through four-byte Unicode) pass one byte at a time, so the payload-offset close fix itself works on macOS. - At most one sporadic
chunk-reuse-*failure per run, always the stale-continuation retirement check.
Any drift in either direction fails the run; a repaired engine that starts passing a documented gap names requalification. Requalify against an adopted upstream revision before retiring a documented gap (the resubmission need is tracked in #536). Bare-metal hosts, remote networks and upstream adoption stay open after macOS CI qualification.
Web WebSocket receive boundary¶
Issue #488 has a separate engine-only browser control. It needs Godot Web
export templates and Playwright Chromium. It uses a local Python server and
WebSocketPeer, with no Signal Fish service or WebRTC extension:
python scripts/check-websocket-web-receive.py \
--godot /path/to/godot --chromium /path/to/chromium \
--output /tmp/websocket-web-receive
The output directory must be new. The server sends 64 exact 4096-byte text messages and a later sentinel. The script holds polling for 100 ms after each burst command, then checks every delivered byte, type and sequence. Results retain commands, engine/export/source hashes, browser version, poll intervals, every callback and every delivered sequence. This opt-in control preserves engine overflow failures outside the deterministic gates.
Use --credit-bytes 32768 for local consumer body-byte batches,
--capacity 524288 for a larger byte buffer, --max-queued-packets 2 to isolate
the packet-count boundary, or --hold-ms 1000 for a longer polling hold.
Credit uses local observations and adds no network acknowledgement protocol.
Zero capacity or packet count keeps the engine default. Keep each control
separate from the original uncredited workload.
For a receipt carried on the tested WebSocket, use
--credit-transport app-ack --credit-bytes 32768. The receiver acknowledges
only the cumulative exact prefix after checking bytes, order and message type.
The server releases that many consumed bytes and keeps all receipts. Each
complete callback body is 4096 bytes; the byte window covers the entire body.
The later sentinel waits for a receipt covering all 64 burst bodies. Use
--ack-delay-ms 100 to hold receipts after consumption. The existing local
batch control remains the default. This experiment needs a cooperating sender;
it adds no delivery guarantee to the SDK or the uncredited workload. The window
must also fit the packet-count limit: on a two-packet queue, a 32 KiB window
still overflows, while an 8192-byte window admits exactly two bodies. Failed
byte or order checks release no receipt or credit. Add --credit-packets 2 to
bound outstanding packets directly; the receiver then consumes and acknowledges
the same bounded batch.
With Chromium 156 on Linux ARM64, all three supported engines deliver 65/65 exact messages with a 32 KiB application window, 100 ms polling holds and 100 ms delayed receipts. No callback is rejected and outstanding bodies stay within 32768 bytes. The matched uncredited 4.7.2 run delivers 31/65 messages; 34 scoped rejections account for every missing receipt. The later sentinel arrives. These local measurements preserve the original overflow verdict.
On a two-packet queue, the byte-only 32 KiB window still overflows: the 4.7.2
run delivers 2/65 messages with six scoped engine rejections. The same byte
window plus --credit-packets 2 delivers 65/65 exact messages with at most two
outstanding packets on all three engines, and no callback is rejected. One
application window must fit both engine limits; this qualifies the tested local
paths, not the uncredited workload or remote networks.
Linux ARM64 and Chromium 156 controls retain these results on Web:
| Engine | Default buffer | 512 KiB buffer | 512 KiB, two packets | 32 KiB body credit |
|---|---|---|---|---|
| 4.3 | Fails | Passes | Fails | Passes |
| 4.4.1 | Fails | Passes | Fails | Passes |
| 4.7.2 | Fails | Passes | Fails | Passes |
Every finite control observes all 65 browser callbacks. The failed byte/count controls retain unique engine rejection identities matching the missing script receipts; the later sentinel is delivered. A larger finite buffer and local credit qualify those controls. The original uncredited receive loss remains open. These are local controls; the remote-path lane below starts the remote qualification.
The receive loss is filed upstream as
godotengine/godot#125073.
A 2026-11-02 rerun on repository main 1915f97e pinned the boundary on
official 4.7.2 Web with default buffers: all 65 browser callbacks arrived,
the engine delivered 16 (sequences 0-14 and the sentinel) and logged 49
scoped rejections; the matched 32 KiB application-ACK control delivered
65/65. Upstream review, repair and adoption remain open.
The weekly WebSocket Web Watch CI workflow (Mondays 10:00 UTC, plus
on-demand dispatch) keeps this boundary under watch on x86-64. It reproduces
the loss on the newest supported official engine, requires the filed
signature (the first 15 burst bodies plus the sentinel delivered, 49 scoped
rejections) and the green 32 KiB application-ACK control, and fails when the
signature moves, flagging either the upstream repair for requalification or
a behavior change for triage (#488). A dispatch input watches other official
releases, such as a dev snapshot after an upstream fix lands.
Remote path qualification¶
The WebSocket Remote Path CI workflow drives the same two controls
over a real remote network path. A credential-free Cloudflare quick tunnel
fronts the fixture server, and the browser reaches it through the Cloudflare
edge over TLS, so every burst frame, command and receipt crosses a remote
path. The fixture options --port N and --public-origin https://host point
the export page and its WebSocket at the public host while the server keeps
its local bind; the report records remote_origin.
Four dispatched CI runs (2026-11-03, ubuntu-24.04 x86-64, Chromium 156, runs 41554600823, 41561638882, 41561939096 and 41562318746; Godot 4.7.2-stable on three of them and 4.3-stable on one, one Cloudflare quick tunnel each) settle the lane. The 32 KiB application-ACK control delivered 65/65 exact messages across every tunnel with all 65 browser callbacks and the full ACK window. Every uncredited run reproduced the filed loss at the payload boundary (godotengine/godot#125073) with the three boundary failures, scoped rejections, a complete trace and partial delivery: 16 of 65 (first gap 15 -> 64), 31 of 65 (15 -> 46), 31 of 65 (30 -> 64) and 16 of 65 (15 -> 64). The exact split varies with network segmentation and engine version, so the workflow pins the stable loss class (the three filed boundary failures, scoped rejections with a complete trace, sub-burst delivery). The weekly schedule (Mondays 12:00 UTC) follows the other watches; dispatch runs collect extra path samples on demand. A same-day pre-merge devcontainer run on Godot 4.3 Web also gave the local 16-of-65 counts.
The same workflow fronts the paced native receive suites
(check-websocket-native-receive.py) with the same tunnel. The driver
options --port N --public-origin https://host pass the fixture options
--port=N and --public-origin=https://host to every selected case: the
fixture server binds the tunneled port and the engine connects wss:// at
the public host. Result records from tunnel runs carry remote_origin.
The lane runs controls and filed defects for each receive issue class (#491
fragmented assembly around control frames, #492 peer reuse retention, #496
full-buffer empty-fragment stall and #498 Unicode close reasons across
reads). Controls must pass over the remote path, with the driver's full
per-case verdict. Defect records must be complete, paced and cleanly
exited. Each defect verdict is pinned per engine to the signature the
collected runs reproduced: the gate keeps a map keyed by engine
major.minor. A pinned defect that passes or fails differently means the
path masked its trigger or upstream moved; requalify the filed issue and
update the map. Dispatching an engine without a map records its defect
outcomes as data; collect runs, then add its map.
Coverage by filed issue, from the same suite on each engine:
| Issue | Filed symptom | Official-engine remote watch | Repaired-engine remote lane |
|---|---|---|---|
| #491 | fragmented bytes change around Ping/Pong (exact receive prefix) |
fails on 4.4.1-4.8-beta2; passes on buffered 4.3, which predates the regression | passes |
| #492 | a fresh peer returns an empty packet after CLOSING reuse (fresh exact bytes) |
fails on 4.4.1-4.8-beta2; passes on buffered 4.3 | passes |
| #496 | a full buffer plus an empty final fragment stalls (complete message missing) |
fails on 4.4.1-4.8-beta2; passes on buffered 4.3 | passes |
| #498 | a valid Unicode close reason becomes 1007 (exact close code, reason and empty packet queue; matching client close frame) |
fails on every collected engine, including 4.3 and 4.8-beta2 | passes |
The 4.3 entry in the #496 row is the paced remote case; the standalone
full-buffer reproduction still drops the following packet on official 4.3.
The repaired legs are the 4.3 backport, 4.4.1, 4.5.2, 4.6.3 and 4.7.2 on
Linux x86-64, plus 4.4.1, 4.5.2, 4.6.3 and 4.7.2 on Windows x86-64
under Wine and 4.7.2 on a native Windows runner. Linux legs run at the
3-byte read cap except the cited uncapped 4.7.2 run; Wine and native
Windows legs run uncapped. The same ladder qualifies the #485 receive
repair. The separate
Gameplay Shutdown Remote lane covers active-shutdown close semantics over
a remote path on official Web 4.3, 4.4.1 and 4.7.2, native 4.4.1 and
4.7.2, and the pinned repaired 4.3 native backport. Each issue stays open for upstream adoption (resubmission
tracked in #536). The macOS map's pinned gaps cover the #492 and #498
families only. Targets with no collected map (bare-metal x86-64 and engine
families not yet collected) also stay open.
A 2026-11-03 pre-merge devcontainer run (official 4.7.2-stable native on Linux ARM64, one Cloudflare quick tunnel, ten selected cases) matched the local 4.7.2 classification exactly: all four controls passed over the tunnel, including the 65-packet burst control, and all six defects failed with their filed signatures. The tunnel runs exposed two fixture races. The pending #492 continuation could flush to the old tunnel connection before the replacement dial, so the retirement check lost its precondition; the fixture now holds the queued continuation until the replacement retires it, which makes the check deterministic on any path (a post-fix tunnel run fails the #492 defect with the filed signature alone). The limit fixture's pong check now waits within the phase budget before judging, because the pong can still be in flight on a remote path when the message completes. The #498 maximum-length mixed close reason also hit a paced-write failure when the engine aborted mid-write in one tunnel run; the gate rejects paced-write errors as unclean samples.
Seven dispatched CI runs (2026-11-03 and 2026-11-04, ubuntu-24.04 x86-64,
official editor binaries, one Cloudflare quick tunnel per native job)
pinned the native map. On 4.7.2 (runs 41591258643 and 41684196081), 4.6.3
(run 41937707094), 4.5.2 (run 41937461420) and 4.4.1 (run 41684522769, whose
native job is green while the run is overall cancelled after its Web job
lost a concurrency race) all four controls passed over every tunnel, with
the 65-packet burst control
delivering every packet (largest delivery gap 283 ms against the 3000 ms
batch deadline). All six defects failed with their filed signatures, apart
from the pre-#541 retirement race in the first 4.7.2 run noted below.
The gate now carries a map for every stable family from 4.3 to 4.7 and
for the 4.8 beta family. The retired-continuation failure in the first 4.7.2 run was the fixture
race #541 removed; post-fix runs fail the #492 reuse defect with
fresh exact bytes alone and retire exactly 2 queued bytes. On 4.3 (runs
41684912691 and 41685515285) the #491/#496/#492 shapes pass: official 4.3
predates those regressions. The #498 close shapes fail with the filed
signature on every collected engine. Same-day devcontainer runs on Linux
ARM64 (4.7.2) matched the CI classification. The weekly schedule (Mondays
12:00 UTC) runs both lanes; dispatch runs collect extra path samples on
demand and add maps for new engines.
Godot 4.8-beta2 is the first collected pre-release family (run
42108167042,
2026-11-05, ubuntu-24.04 x86-64, official editor, one Cloudflare quick
tunnel per job). Both lanes follow the 4.4-4.7 shape. Native: the four
controls pass, including the 65-packet burst, and the six defects fail with
their pinned signatures, so the gate carries a 4.8 map. Web: the 32 KiB
application ACK control delivers 65/65 with the full 64-packet ACK window,
while the uncredited run delivers 16/65 with 49 scoped rejections (first
gap 15 -> 64), the same loss class as godotengine/godot#125073. The
installer fetches pre-release builds from godotengine/godot-builds; stable
releases keep using the main repository.
The WebSocket Native x64 lane extends the same tunnel idea to the
experimental source-build repair (#485). With remote_path on it builds the
pinned 4.7.2-stable source twice on ubuntu-24.04 x86-64, repaired via
scripts/repair-websocket-stable.py and unpatched, enables the mbedtls
module so the engines can speak wss://, and fronts the fixture server with
a Cloudflare quick tunnel. The repaired engine runs the ten pinned receive
cases over the tunnel at the lane's forced 3-byte read cap on the default
Linux dispatch; the baseline runs uncapped, the conditions the
official-engine pinned signatures were collected under. The repaired engine
must pass every case. The baseline must keep its pinned map: each
expected-failure case must keep failing with its filed signature, and each
expected-pass case must keep passing, so neither a pass nor a drift can
come from the path masking a trigger. The pinned 4.3 map expects only its
two #498 close shapes to fail;
official 4.3 predates the other four defect shapes, and the gate requires
those four to keep passing on the 4.3 baseline. That
requirement applies to a Linux engine family with a pinned remote-path map;
Windows and unpinned families record defect outcomes as collection data. The
gate checks binary identity by SHA-256, tunnel provenance, paced writes,
clean exits and per-case verdicts, keyed by engine family for the baseline
map. A baseline defect sample may contain engine errors (for example
the #492 reuse case's PacketBuffer ERR_BUG line); the exit,
paced-write, record and pinned-signature checks still bound it.
Pass-required cases, every repaired case and each baseline control,
must stay free of engine errors, which the driver's verdict already
enforces.
Three dispatched runs (2026-11-03, 1d2e7604, official 4.7.2-stable
importer, one quick tunnel per variant) settle the first remote-network
qualification of the repair. In the capped run
(41700349763)
the repaired engine (4b58fac581f2b2f293d85b2eb313bf59d9fc227a9d121f3fbaeb21bfd7ae6130)
passes all ten cases over the tunnel with every read capped at 3 bytes; the
65-packet burst control delivers every packet with a largest delivery gap of
255 ms against the 3000 ms batch deadline. The unpatched baseline
(77bbcc386e6415a611d827051d24edd82b7eb9f44fb858ab57a4413405c8144e) from the
same source and flags passes its four controls and fails all six defect
shapes with the exact pinned signatures on its own tunnel. In the uncapped
run (41700353810)
the repaired engine passes all ten cases uncapped (burst largest gap
276 ms). Its baseline run reproduced five of six pinned signatures exactly
and then hit the documented mid-write abort on the maximum-length mixed
close shape (paced socket write failed); the gate rejects paced-write
errors as unclean samples, and the re-dispatch
(41706656767)
collected the clean sample: the baseline fails all six pinned shapes again.
The abort stays a documented one-off, as in the official-engine lane. The
first dispatch attempt
(41697078313)
failed every remote connect with WSS is not available in this build: the
minimal source build shipped no TLS module. Its repaired engine still passed
the full local paced suite, 184 of 184 cases at the 3-byte cap. Dispatch
runs collect extra remote samples on demand; the cap applies on the default
Linux dispatch, so a Windows or zero-cap remote dispatch qualifies the
repaired leg uncapped.
The pinned 4.3-stable backport
77dcf97d82cbfe4e4615475fa52ca03da645dbd8 completes the same remote leg
(run
41734479410).
The repaired engine (8aebdbce741747e08ba00ec694764e38d9d311ee94e5193d582c447197d831f3)
passes all ten pinned cases over its Cloudflare tunnel with every read
capped at 3 bytes; the 65-packet burst control delivers every packet with
a largest delivery gap of 248 ms against the 3000 ms batch deadline. The
unpatched 4.3 baseline (68c4bce9bca9c3947428069ab1d30ad5c2c2ebd6ee5c7acbef9749f293fc70df)
on its own tunnel passes its four controls and the four pre-regression
shapes, and fails exactly the two #498 close shapes with the pinned close
signature, matching the official-engine 4.3 watch maps. That closes the
4.3-backport leg of the #485 remote-network qualification.
The pinned 4.4.1-stable repair
49a5bc7b616bd04689a2c89e89bda41f50241464 completes its Linux x86-64
remote leg (run
41754464659).
The repaired engine
(7531c11dc69503e334d9048d6dbdb873f920942988a0d6fa58d95866ca8ec900)
passes all ten pinned cases over its Cloudflare tunnel with every read
capped at 3 bytes; the burst control delivers 65 of 65 packets with a
largest delivery gap of 248 ms against the 3000 ms batch deadline. The
unpatched baseline
(e082eb387fa1f3e008c2967712fc644a770e31ad99bd6f363eb368f16aadb5dd) on
its own tunnel passes its four controls, fails all six pinned defect
signatures, and delivers its burst 65/65 (largest gap 262 ms). A first
dispatch (41751187507)
had already qualified the repaired leg; its baseline sample stayed
unclean (the documented mid-write abort) while the lane
still rejected engine-error defect samples outright, so the gate now
tolerates a defect's filed symptom and the re-dispatch collected the
clean sample above.
The pinned 4.5.2-stable repair
6ce3de25aa58466e14ef354703ba8d9791a417da completes the same leg
(run
41754468768).
The repaired engine
(f158c982bff211daf0c41e197d79f90b8f09acddf99f0f55d0015162ac6ce50f)
passes all ten cases at the 3-byte cap (burst 65/65, largest gap
255 ms). The unpatched baseline
(4defc7d31eab80f57ee4abb07ba09ce90b2ef6a23df079ca62e3980ba1a8151f)
passes its four controls and fails all six pinned signatures on its own
tunnel, so the lane pins the 4.5 family map to those six signatures.
The pinned 4.6.3-stable repair
35e80b3a8822a9df9be390814b62f44c0a9c69e8 completes the same leg
(run
41754472264).
The repaired engine
(ce17e58543d06978d479d7568bd561713f022cdbafc85948f5cbef37a7fe5210)
passes all ten cases at the 3-byte cap (burst 65/65, largest gap
269 ms). The unpatched baseline
(7ab2a67a48b4266743d2e05acf0a4db0551deba2117f8e7bf55e6dd7b2af1714)
matches the 4.5 baseline outcome set and the lane pins the 4.6 family
map the same way. That closes the 4.4.¼.5.2/4.6.3 stable-repair
remote legs of the #485 remote-network qualification.
The Windows dispatch closes the last planned #485 remote leg. In the
first run (41754475885)
the repaired 4.4.1 engine
(3e7f09f711eac24df914e549762a92e1cf10127f6c096bf0b555f506ac949e6f)
passes all ten pinned cases over its Cloudflare tunnel uncapped under
Wine, with the burst control at 65/65 packets (largest gap 257 ms).
The unpatched Wine baseline
(d5b78c0fe2940d0874fb0b488303a88157192fbb5858b93ceafa91e33eb1d75b)
records clean outcomes on nine of its ten case samples (four controls
pass, five defect shapes fail with the pinned signatures) but hits the
documented
mid-write abort on close-3-mixed-server, so the gate rejects the
sample and a re-dispatch
(41763583302)
collects the clean one: the repaired engine
(42cb1f63f4ae64b47e9487a1246e5d57bf51a1081a7bbfa411ad605f25d8c7c1)
passes all ten cases again (burst 65/65, largest gap 248 ms) and the
baseline (fe39d6ccf1f79285f45afc0883953c3bf631c50165348f0c10fc6b430c4ab9f4)
passes its four controls and fails all six pinned signatures, its
close-3-mixed-server sample now clean. The second collected Wine
sample (41774059122)
confirms the map: the repaired engine
(4905f4af3ea7c005a82aed0ddeddb9af8c91a72aa2a508a9ae138ee60832bd19)
passes all ten cases again (bursts 65/65, largest gap 283 ms) and the
baseline (38c0e2580628c19117bf47dc9ca9cec7b56883012bff3882bb626b583e5beb45)
passes its four controls and fails the same six pinned signatures with
clean close samples (bursts 65/65, largest gap 909 ms against the
3000 ms batch deadline, below the recorded 1.14 s benign Wine stall).
The gate pins the Wine 4.4 family map from these two runs; the other
families stay collection data on Wine until their own collected runs
pin maps.
The test-only observer wraps the engine's assigned browser onmessage handler.
Each rejection records the physical connection, callback ordinal, byte length,
bounded prefix, rejection reason and browser clock while that handler is active.
Payload-capacity and packet-count rejections stay distinct. An error outside
the handler has no callback identity. Callback arrival alone does not prove
engine acceptance. A complete trace must account for omitted records before
comparing rejected identities with script receipts.
The active-gameplay driver also accepts these comparison controls:
--relay-packets-per-second 240: use elapsed time to set the offered target, with at most 16 sends per frame to catch up. Zero keeps the original frame rate.--relay-credit-bytes 32768: cap outstanding 4096-byte bodies using the other local seat's receive count. JSON framing and other protocol traffic add bytes.--relay-credit-packets 2: cap outstanding relay packets the same way. Combine it with a byte window so the application contract fits both engine limits.--receive-pause-ms 100: hold engine polling after each mesh generation opens. Other application and mesh work continues. Zero leaves polling unchanged.--inbound-buffer-bytesand--max-queued-packets: set the engine capacities. Zero preserves each engine's default. A Web property value does not establish the browser's backing queue capacity.
Add --relay-credit-transport app-ack --relay-credit-bytes 32768 for an
application ACK comparison. Each receiver sends a cumulative exact-prefix
receipt through GameData. Only a validated receipt from the current remote
player, room and gameplay generation advances the sender's byte window.
A missing or corrupt data packet stops new ACK progress. The default local
credit mode keeps the shared-process receipt counter. ACK records retain
physical connections, sent/received prefixes and window peaks. A packet window
from --relay-credit-packets is enforced on both seats and recorded beside the
byte-window peaks.
Application ACKs add protocol traffic and change the transmitted schedule. This is an opt-in drill control, with unchanged 4096-byte data bodies. It establishes no SDK-wide delivery guarantee. Preserve the original uncredited failures and qualify real remote paths separately.
On Linux ARM64, the 32 KiB application ACK control with a two-packet window passes the Web 4.7.2 stalled-close case against the released v0.12.0 server with 50 ms TCP delay, 127-byte proxy writes and a 100 ms initial polling hold. Both seats peak at two outstanding packets, keep exact relay prefixes, exact mesh receipts in all three modes and the required nonresponding close outcome.
On Linux ARM64, the 32 KiB application ACK control passes all three shutdown cases on native Godot 4.3 and Web Godot 4.4.¼.7.2 with 50 ms TCP delay, 127-byte proxy writes and a 100 ms initial polling hold. All exact mesh, cached retirement and fresh-gameplay checks pass. A matched uncredited Web run retains a scoped rejected callback and relay gap from 151 to 152; both seats still preserve the semantic shutdown close. The engine receive defect remains open. The application ACK control qualifies its bounded local workload.
Logs identify the seat and physical connection, reported capacities, poll intervals and duration, queued packet peaks, total received frame bytes, first relay gap, offered target, credit blocks from either window, and peak outstanding body bytes and packets. All original shutdown, cached-packet and exact mesh checks remain required. Keep rate, credit, pause and capacity changes separate in reports.
A 240-packet/s native 4.3 run with a 100 ms poll hold fails on an engine overflow and a first delivered gap from 15 to 24. Its matched 32 KiB body-credit control passes all gameplay checks. The credit result qualifies that local control; it leaves the original uncredited workload failed. A new six-case Godot 4.4.1 native/Web cohort passes graceful shutdown, reset, nonresponding close, cached metadata, retirement and fresh mesh recovery at the original four-per-frame rate. Three new Godot 4.3 Web shutdown controls also pass, and the 4.3 Web shutdown leg is now qualified over the remote path. Godot 4.3 native receive qualification remains open.
Delayed relay shutdown¶
Server v0.12.0 release qualification¶
The published Linux ARM64 server v0.12.0 archive was checked against its
published SHA-256. Its source revision is
92842406.
The server executable SHA-256 is
90d8cd0799ee0e61e20cfbf2da8e5996811c100f7b794560bdd48676fac110d1.
With 50 ms TCP delay, 127-byte proxy writes, a 100 ms polling hold and the original four 4096-byte bodies per frame:
- Repaired native Godot 4.3 passes graceful, reset and stalled-close recovery.
Every mode preserves exact relay prefixes, cached mesh retirement and fresh
mesh gameplay. An earlier graceful attempt exhausted the native send queue
and lost the semantic close. Issue #510 adds recoverable
ERR_BUSYhandling. - Repaired native Godot 4.7.2 also passes all three shutdown modes.
- Official Web Godot 4.7.2 on Chromium 156.0.8078.4 passes graceful shutdown. Its reset and stalled-close runs retain interior relay gaps. Browser socket 2 rejects sequences 56-59 in four 4229-byte callbacks; socket 4 rejects 34-35. These are scoped engine callback rejections. Issue #488 remains open.
- The matched Web application ACK control (
--relay-credit-transport app-ack --relay-credit-bytes 32768) passes all three modes with no callback rejection. This window counts body bytes. Other protocol traffic adds bytes.
Those Web rejection captures preceded the native send-capacity change. A later cohort passes all three Web modes with and without ACK credit. After the final native capacity update, another uncredited stalled-close control again fails: socket 1 rejects five 4229-byte callbacks for sequences 35-39, and delivery jumps from 35 to 40. Graceful and reset controls pass in that cohort. Passing schedules leave these failures unresolved. The native capacity change does not repair Web receive queues.
Reports separate relay and ACK busy attempts. The drill checks that both stop after terminal state. During a valid observed close it permits one application refusal and one separate transport-status refusal. Close metadata, delivered prefixes, retirement and recovery remain strict.
Join-only admission also passes on official Godot 4.3, 4.4.1 and 4.7.2 for
both protocol versions. Each native engine passes the real WebSocket smoke
with a 16-byte outbound limit: oversized UTF-8 text and binary frames return
ERR_INVALID_DATA, an exact-capacity frame is delivered, and the close completes.
All attempts, commands, hashes and reports are retained locally under
/tmp/session470/ and progress/session470-artifacts/. Setup failures and
attempts that imported intermediate source remain separate. These are local
Linux ARM64 controls; remote networks, other targets and upstream engine
adoption remain open for #482, #485 and #488.
Delay controls¶
Add --network-delay-ms 50 to the active-shutdown command to delay each TCP
direction by 50 ms. This includes relay data and signaling. Mesh UDP and HTTP
control stay local. The workload, close deadlines and exact gameplay checks
stay unchanged. This is local network emulation; real remote paths need their
own qualification.
Use --network-chunk-bytes 127 to bound each proxy read and write. TCP may
coalesce these writes before Godot reads them. This does not promise a matching
engine read size or WebSocket frame fragmentation. Queue slots use a 1 MiB
payload budget per direction, capped at 8192 chunks. Reports retain the
effective byte/count bounds, bytes, chunk counts, peak queue size and observed
delay for each physical connection. A full queue can add TCP backpressure.
EOF follows queued data. Socket errors and cleanup cancellations stay visible.
Each passing case must forward both directions on at least four connections
and meet the requested minimum delay.
The proxy's loopback integration tests are opt-in:
PYTHONPATH=tests python -m unittest test_delayed_tcp_proxy. They cover large exact
binary streams, half-close replies, actual delay, reset and idle teardown.
Browser logs include socket state reads and close callbacks. On the original 4.4.1 delayed run, both browser sockets were CLOSING while the engine reported OPEN. A send returned FAILED; the SDK discarded the peer before close 4000 arrived about 80 ms later. The transport now refuses further sends and polls for that callback for at most one second, then falls back to one failure. Text and binary refusals use the same rule. All strict receipt, shutdown, cached metadata, retirement and fresh-gameplay checks remain required.
One repaired 4.4.1 native/Web 50 ms cohort passes all six shutdown cases. Native 4.3 still fails exact relay receipts in all three delayed controls; its matched 32 KiB local body-credit controls pass. The 4.7.2 small-write cohort preserves shutdown and mesh checks, but its Web graceful case drops relay packets after restart (#488). Both seats expect sequence 24 and receive 26/27 after engine buffer errors. Its matched 32 KiB body-credit cohort passes all three cases. Later final-source 4.4.1 Web graceful and reset runs also fail on receive overflow. The graceful gap is from sequence 535 to 536 during the first generation; its close, retirement and fresh mesh remain exact. Credit changes the local offered work; it does not repair or qualify the original uncredited receive boundary. Every failure stays retained.
Add --trace-receive to retain scoped browser callback and rejection records
for the active-shutdown drill. The driver freezes the observer source per
cohort, copies it per case, and records its hash. Generated relay bodies also
get a parsed relay_sequence; other messages leave it null. Snapshots are
collected before the browser closes, including failed controls.
A new uncredited 4.7.2 Web run with 50 ms delay and 127-byte proxy reads fails with a scoped rejection and a generation-one relay gap from 21 to 22. A separate 100 ms polling-hold control identifies rejected relay sequence 23 and the matching application gap from 23 to 24. Both retain GoingAway, one close 4000/server_shutdown per seat, retry eligibility, cached retirement and exact fresh mesh receipts. Their overall verdicts stay failed. Matched 32 KiB body-credit controls pass. Passing attempts and every failure stay separate; this evidence leaves engine repair and remote qualification open.
Controlled lifecycle publication¶
tests/smoke/run_controlled_lifecycle_acceptance.gd adds local server barriers
for #356, #358 and #367. Use an isolated server checkout at
182cb234c7aa601be3c2aad881948b3c5b853aaa, the config above, and Rust 1.94.0.
Apply only tests/fixtures/controlled_lifecycle.patch for this drill.
# From the isolated server checkout:
git apply /path/to/signal-fish-client-godot/tests/fixtures/controlled_lifecycle.patch
cargo build --locked --bin signal-fish-server
export SF_GODOT_BARRIER_DIR="$(mktemp -d)"
printf '%s\n' "$SF_GODOT_BARRIER_DIR"
SF_GODOT_BARRIER_DIR="$SF_GODOT_BARRIER_DIR" target/debug/signal-fish-server
# In another terminal, from the Godot client checkout, set the printed path:
export SF_GODOT_BARRIER_DIR=/path/printed/above
godot --headless --path . --script tests/smoke/run_controlled_lifecycle_acceptance.gd
Run one drill per server and barrier directory. The server consumes .arm
files, writes .reached files and waits for .release files. Each server
wait has a 15-second limit. Each client wait has an eight-second limit and
the full drill has a watchdog. Client cleanup releases held publication
barriers. After testing, stop the server, remove the barrier directory and
reverse the patch with git apply -R. Rebuild before using that checkout
for ordinary server runs. The patch is acceptance instrumentation.
The drill checks these new boundaries on Godot 4.3 Linux arm64:
-
367: two real reports from one socket call the public server dispatcher¶
concurrently. The first pauses after state acceptance and before fanout. The second reaches the report lock before release. Both true/false orders reach the v3 observer in acceptance order; v2 receives no transport events. Server assertions under the report lock check stored state at each queue admission. A duplicate is suppressed. Both observers receive later gameplay. The ordinary transport-report drill above covers reconnect generations. -
358: a Direct metadata transaction pauses inside the commit hook before¶
storing the endpoint. Join and leave each reach the room mutation lane while it is held. No plan or membership change appears before release. Clients converge on the fresh endpoint and generation; plan callbacks see peers in the current roster, and relay gameplay succeeds afterward. -
356: the first registration attempt returns failure at the existing result¶
seam. A successful retry preserves terminal identity, reason, roster removal, replay and fresh gameplay. A panic on retry exhausts publication and closes the viewer once with1000/unregistered. No terminal acknowledgement is sent through the unregistered generation. Room members still receive one removal. Fresh authentication and spectator admission restore gameplay. Both recovery and exhaustion run with v2 and v3 viewers.
The server's normal WebSocket reader dispatches serially. The fixture starts transport-report tasks through the public library entry point to reproduce embedded-host concurrency. It preserves the production report, lifecycle, room and queue gates. The injected registration failures qualify recovery behavior at that seam; the drill does not establish a request-triggered coordinator failure. The publication drill below adds authoritative spectator removal, correlated result decoding and controlled Direct reconnect publication. Browser interop is separate.
Gameplay publication boundaries¶
Using the same patched server and barrier directory, run:
This drill passed with Godot 4.3 Linux arm64, client base 5abec3c and
server 182cb234 plus the checked-in controlled_lifecycle.patch:
-
368: pause spectator admission before baseline registration while holding¶
the room mutation lane. A player join reaches that lane and waits. Gameplay continues to an incumbent player. The new viewer receives no traffic before publication. After release, it receives one baseline then one player delta, a complete roster and fresh gameplay. Prior gameplay is absent from the new viewer's history. The drill covers v2/v3 JSON and raw MessagePack. -
356: an administrative detach with reason
first registration failure. v2/v3 viewers receive one terminal with the original room identity and reason. A connected member receives one removal; an absent v3 player receives that removal once in decoded replay. Replay requires consumer dispatch. Readmission and gameplay succeed.removedrecovers after the¶ -
356: a fixture sends one raw v3
negotiatingRoomOperation/LeaveSpectatorafter¶room_operation_ids. Recovery preserves operation identity,SpectatorLeftpayload, room identity, reason and roster. The SDK emits one informational result and leaves its cached session unchanged. The server removes the viewer from routing. This fixture adds no room-operation API or admission support. The ordinary SDK usesleave_spectator(). -
358: pause a Direct endpoint commit while reconnect reaches the mutation¶
lane. Two reconnect cycles receive their baseline before fresh plans, rotate the credential and converge on the new endpoint and generation. A game-ownedStreamPeerTCPconnects to a real loopback listener. It sends payloads after reconnect and running endpoint refresh. Service disconnect, endpoint replacement and Relay reset close the prior socket; the listener observes EOF. Relay gameplay continues after teardown.
The added barriers are spectator-baseline and reconnect-enter.
spectator-remove.arm makes the next spectating client's Ping call the
existing administrative detach path with reason removed, then process the
Ping normally. Combine it with detach-error.arm for recovery. This trigger
is local acceptance instrumentation; it is absent from ordinary builds.
The mutation, routing, registration retry and publication paths remain intact.
The drill releases held barriers and closes application sockets on failure.
All waits and the overall watchdog retain the bounds described above.
The pressure drill below covers server delivery pressure. The roster recovery drill covers observer metadata. This native TCP fixture provides no browser Direct-socket contract; browsers require supported transports.
Server delivery pressure¶
tests/smoke/run_delivery_pressure_acceptance.gd covers #368 at the server
outbound queue. Use the isolated server, config and toolchain above. Apply
controlled_lifecycle.patch, then tests/fixtures/delivery_pressure.patch.
Set websocket.send_queue_capacity to 4 and keep
websocket.slow_consumer_timeout_ms at 5000. Rebuild the server and start
it with a fresh SF_GODOT_BARRIER_DIR, shared with the client process:
The added barrier is writer-<spectator-id>. After writing a control frame,
the server writer waits before selecting another queue entry. The client
arms only its spectator's writer and wakes it with Ping. The sender's Pong
then proves that the server processed every ingress frame while the writer
was held. The production queue admission, coalescing and reporting paths
remain active. The drill releases the writer on failure. Reverse both patches
and restore the queue config after testing.
- v3 JSON latest: eight updates to one key produce the final value, exact
latest_supersededranges for sequences 1-7 and matching typed counters. - v3 JSON volatile: a full four-entry queue evicts sequences 1-4 and delivers
the newest four values in order. Exact
volatile_droppedranges and typed counters match. Both lossy cases publish the report before surviving data, preserve the spectator role and receive fresh reliable gameplay afterward. - Reliable: five frames fill the four-entry queue and trigger one spectator close with code 4002. The sender stays in its room. The viewer clears its state, authenticates on a fresh connection, rejoins and receives fresh data. The matrix covers v2/v3 JSON and raw/decoded MessagePack, plus v3 opaque rkyv/protobuf. Opaque test bytes remain application-owned.
Latest and volatile sends use a raw v3 text fixture. The public SDK send
methods currently use reliable delivery. This drill adds no lossy-send API
or binary delivery-metadata API. The v2 verdict applies to the pinned server
build; v2 has no spectator_relay capability token.
Roster recovery contract¶
tests/smoke/run_roster_recovery_acceptance.gd is an opt-in regression for
issue #374. It needs the ordinary fixed server above, with no instrumentation:
Use server
b264ec7
with the ordinary config above. Godot 4.3 Linux arm64 passed six cases: v2/v3
observers with JSON and raw/decoded MessagePack. A seated observer, an incumbent
spectator and a spectator admitted during absence restore the current name,
readiness and authority before each reconnect callback. Each case includes two
reconnect cycles, exact resumed gameplay, spectator role and terminal leave.
Reconnect emits no synthetic player join. Success prints
LIVE_SERVER_ACCEPTANCE_PASS.
Server 182cb234 omits the metadata and fails this drill. The new single-frame
contract shipped in
server PR #1169.
The deterministic client suite also covers missing/malformed metadata,
ID/epoch/sequence matching, duplicates, older epochs, replay baseline precedence,
room reset, bounded roster/departure history and mesh incarnation retirement.
Expired reconnect cleanup¶
tests/smoke/run_expired_cleanup_acceptance.gd covers issue #357. In an isolated
server checkout at b264ec7, use the config above with
server.reconnection_window set to 1. Apply this repository's
tests/fixtures/expired_cleanup.patch, then rebuild:
# From the isolated server checkout:
git apply /path/to/signal-fish-client-godot/tests/fixtures/expired_cleanup.patch
cargo build --locked --bin signal-fish-server
export SF_GODOT_EXPIRY_DIR="$(mktemp -d)"
SF_GODOT_EXPIRY_DIR="$SF_GODOT_EXPIRY_DIR" target/debug/signal-fish-server
In another terminal, set SF_GODOT_EXPIRY_DIR to the same directory and run:
A Ping with prepare.arm captures the server regression's controlled residue:
the server removes a seat while retaining its socket, registers a pending
reconnect, waits for expiry and snapshots its real cleanup candidate. It then
sends RoomLeft and Pong. The client joins the same room with the same socket
identity and receives a new RoomJoined credential. A Ping with cleanup.arm
runs the saved candidate through the production cleanup method before Pong.
The cleaned file reports zero removed seats.
Godot 4.3 Linux arm64 passed four cases with v2/v3 observers and JSON/decoded MessagePack. The fresh seat becomes ready, sends exact gameplay and reconnects with its current credential. An ordinary expiry control refuses the expired membership and permits a fresh login/join. The actor uses v3 because this server pre-issues reconnect credentials on v3. V2 observer results qualify this tested server build. They do not imply v2 credential issuance.
The fixture forces a server-side residue seam. It does not establish that ordinary remote messages can create the stale record. Candidate capture has a four-second deadline; client phases wait eight seconds, and the runner has a two-minute watchdog. Run one drill per server and directory. Stop the server, remove the directory, restore its config and reverse the patch after testing. The fixture is local acceptance instrumentation.