ARM capacity campaign: audit and experiment ledger¶
Campaign: C0–C5 (#636).
This ledger starts at b24b5e13 (2026-09-27). It inventories code and
existing evidence. Inventory is not a correctness review or a capacity
measurement. Keep a row open until its invariants and missing cases have been
checked against the named revision.
Audit method¶
Allowed states are unreviewed, hypothesis, reproduced, fixed, and deferred. A deferred finding needs a linked issue. A reproduced finding needs a deterministic test or equivalent direct proof. A fixed finding needs the code revision and regression check. Existing green tests alone do not advance a row. For each slice, inspect the named code, enumerate failure paths, write a red test for any defect, sweep related paths, and record the green result. Keep correctness findings separate from performance hypotheses.
Finding records use this form; assign IDs ARM-C001, ARM-C002, ... for
correctness and ARM-P001, ARM-P002, ... for performance:
| Field | Required content |
|---|---|
| ID, state, severity | Stable ID; one allowed state; critical/high/medium/low |
| Player impact | Concrete gameplay or resource effect; affected feature |
| Source and revision | Path, function/line, inspected commit |
| Invariant | Expected behavior and the observed violation or hypothesis |
| Confidence and reproduction | Direct proof or exact failing command, seed, setup, and output |
| Disposition | Fix/test commits or a linked issue with mitigation and owner |
No new finding is confirmed by this initial inventory.
Correctness findings¶
ARM-C001 — Spectator cap during room creation¶
| Field | Record |
|---|---|
| State, severity | Fixed, medium |
| Player impact | Spectators can enter a new room before its configured cap is set. This can exceed the cap and grow spectator fan-out. |
| Source and revision | src/database/mod.rs room creation and src/server/room_service.rs cap application; src/server/spectator_service.rs::join_owned, reviewed at 684ca906. |
| Invariant | A spectator must see the configured cap before admission. The room row is visible with max_spectators=None while the creator still holds the room-code lock. The old spectator path read that row without the lock. |
| Confidence and reproduction | Deterministic paused-create test: scripts/dev-loop.sh spectator_join_waits_for_creation_cap_before_admission failed before the fix with spectator admission must wait until creation applies its cap. |
| Disposition | The spectator path now holds the room-code lock until it enters the room event lane, then reads the room again before admission. The paused-create and lock-failure tests pass. Fix revision: 38e2e42d. |
ARM-C002 — Spectator admission on a retired room code¶
| Field | Record |
|---|---|
| State, severity | Fixed, high |
| Player impact | A spectator can enter a room with its old code after the authority rotates that code. |
| Source and revision | src/server/spectator_service.rs::join_owned and src/server/moderation.rs::handle_regenerate_room_code_operation, reviewed at 5a48ee29. |
| Invariant | Rotation must finish after any old-code admission already in progress. The spectator path releases the old-code lock once it enters the room event lane; the old rotation path changed the code without entering that lane. |
| Confidence and reproduction | Paused in-lane room read: scripts/dev-loop.sh rotation_waits_for_in_flight_old_code_spectator_admission failed before the fix with rotation must wait for the admitted old-code spectator. |
| Disposition | Rotation now holds the room event lane across the storage swap, after acquiring both code locks. The regression checks old-code refusal, new-code admission, roster, and lock release. Fix revision: 8a22035b. |
ARM-C003 — Rotation waits on its own or a busy code lock¶
| Field | Record |
|---|---|
| State, severity | Fixed, medium |
| Player impact | A generated code equal to the room's current code makes rotation wait on its own lock and fail. Two rotations can wait on each other's old codes. |
| Source and revision | src/server/moderation.rs::handle_regenerate_room_code_operation, reviewed at 5a48ee29. |
| Invariant | Candidate selection must make bounded progress while the old-code lock is held. The old path waited for each candidate lock, including its own. |
| Confidence and reproduction | Scripted OLDCOD, then NEWCOD: scripts/dev-loop.sh regenerate_skips_own_or_busy_code_before_waiting_for_a_second_lock failed before the fix with rotation must skip OLDCOD without waiting. The test also holds a candidate lock to model the cross-rotation wait. |
| Disposition | Rotation skips its own code and busy candidate locks within the existing eight-attempt budget. Both scripted cases pass. Fix revision: 8a22035b. |
ARM-C004 — Spectator admission after shutdown drain¶
| Field | Record |
|---|---|
| State, severity | Fixed, medium |
| Player impact | A spectator can join during shutdown after waiting for a room lock or response capacity. The new role is short lived and can reach peers during teardown. |
| Source and revision | src/server/spectator_handlers.rs::handle_join_as_spectator_operation and src/server/spectator_service.rs::join_owned, reviewed at 1e42cf07. |
| Invariant | Shutdown drain refuses new spectator roles. The handler checked drain only before the owned admission, which can wait at the room-code lock or baseline delivery. |
| Confidence and reproduction | The initial spectator_join_waiting_on_room_code_refuses_shutdown_drain test failed before the fix: the join returned SpectatorJoined after drain. Its deterministic replacement is scripts/dev-loop.sh spectator_join_rechecks_drain_after_code_lock; companion tests trigger drain after storage add and during a blocked baseline. |
| Disposition | The service rechecks drain inside the room lane, rolls back a durable add if drain starts during storage, and cancels a blocked baseline through the drain signal. The focused tests pass in this change (#647). |
ARM-C005 — Reconnect admission after shutdown drain¶
| Field | Record |
|---|---|
| State, severity | Fixed, medium |
| Player impact | A reconnect can restore a seat and spend its one-time token after shutdown drain starts. The forced close then removes that short-lived seat. |
| Source and revision | src/server/reconnection_service.rs::handle_reconnect_owned and src/server.rs::register_local_client_with_initial_message_async, reviewed at 07952b86. |
| Invariant | Drain refuses reconnects until the Reconnected baseline is queued. Rejection rolls back the seat and releases the token. A queued baseline commits the reconnect. |
| Confidence and reproduction | reconnect_restoring_membership_refuses_shutdown_drain_and_releases_token failed before the fix: a paused membership write resumed after drain and returned success. reconnect_waiting_for_baseline_capacity_refuses_shutdown_drain covers the blocked response queue. |
| Disposition | Reconnect rechecks drain after the room gate and durable add, cancels a blocked baseline when drain starts, and serializes baseline enqueue with the drain transition. Both regressions pass (#647). |
ARM-C006 — Reconnect can restore a duplicate player name¶
| Field | Record |
|---|---|
| State, severity | Fixed, medium |
| Player impact | Another player can join with a disconnected player's name, then the old player can restore the same name through reconnect. |
| Source and revision | src/server/room_service.rs join name validation, src/server/reconnection_service.rs restore, and src/database/mod.rs::add_player_to_room, reviewed at 07952b86. |
| Invariant | Seated player names must remain unique under the join path's canonical comparison. |
| Confidence and reproduction | reconnect_name_taken_by_new_member_rejects_without_spending_token failed before the fix: reconnect restored Straße beside a seated STRASSE. |
| Disposition | Reconnect now checks the saved name under the room event gate before restoring membership. A conflict returns ReconnectionFailed and leaves the token usable until its window expires (#647). Capacity and names are not reserved during disconnect. |
ARM-C007 — Room creation can publish after shutdown drain¶
| Field | Record |
|---|---|
| State, severity | Fixed, medium |
| Player impact | A creator can receive RoomJoined after shutdown drain starts and enter a room that is about to close. |
| Source and revision | src/server/room_service.rs::handle_join_room_owned and its async baseline commit, reviewed at fd18c6dc. |
| Invariant | A new room must not commit its first join after drain starts. Existing-room joins remain allowed during drain. |
| Confidence and reproduction | scripts/dev-loop.sh draining_room_creation_cancels_baseline_before_it_is_queued failed before the fix: a paused baseline room read resumed after drain and delivered RoomJoined. The test covers explicit and generated room codes. |
| Disposition | Created-room baseline enqueue now shares the shutdown drain commit gate. A canceled baseline rolls back the unpublished room and returns SERVER_DRAINING (#647). |
ARM-C008 — Creator name write failure can publish the placeholder name¶
| Field | Record |
|---|---|
| State, severity | Fixed, medium |
| Player impact | A room creator requesting a display name can receive RoomJoined with the stored placeholder Creator if the name write fails or reports a missing row. |
| Source and revision | src/server/room_service.rs creator admission and src/database/mod.rs::update_player_name, reviewed at 6877e4b0. |
| Invariant | A successful creator join publishes the requested name. A failed name write refuses the join and releases the unpublished room and its code. |
| Confidence and reproduction | creator_name_failure_refuses_creation_and_allows_retry failed before the fix with RoomJoined and Creator; creator_name_missing_row_refuses_creation_and_allows_retry covers Ok(false). Both verify room rollback and a successful retry with the requested name. |
| Disposition | Creator admission now requires a confirmed name write. On storage failure or missing row it rolls back the unpublished room and returns ROOM_CREATION_FAILED (#647). |
ARM-C009 — Failed spectator-cap write can publish an unlimited room¶
| Field | Record |
|---|---|
| State, severity | Fixed, medium |
| Player impact | A storage failure while setting a new room's spectator cap can leave it unlimited despite a positive deployment cap. Spectators can then exceed the configured limit. |
| Source and revision | src/server/room_service.rs creation-time cap write and src/database/mod.rs::set_room_max_spectators, reviewed at 6d2b3eb2. |
| Invariant | A creator join cannot publish a room with a cap different from the deployment policy. Explicit 0 is the unlimited opt-out. |
| Confidence and reproduction | spectator_cap_write_failure_refuses_creation_and_allows_retry failed before the fix with RoomJoined. The green test checks room/code rollback, retry, and one-spectator enforcement. explicitly_unlimited_spectators_need_no_cap_write covers the opt-out. |
| Disposition | A failed positive cap write refuses creation. The in-memory backend now hides the pending room and retries deletion if rollback fails (ARM-C013). Other backend and lifecycle cases remain in #658. |
ARM-C010 — Old peer metadata can overwrite a rejoined seat¶
| Field | Record |
|---|---|
| State, severity | Fixed, medium |
| Player impact | Peers can receive a former endpoint at game start after a player leaves and rejoins the same room. |
| Source and revision | src/server/game_data.rs::handle_provide_connection_info and src/database/mod.rs::update_player_connection_info, reviewed at 5345b9ec. |
| Invariant | A metadata write from the old seat must finish before that player can leave and rejoin. The old handler read the room, then awaited storage without the client lifecycle gate. A new seat with the same player ID could receive the old write. |
| Confidence and reproduction | scripts/dev-loop.sh old_connection_info_cannot_overwrite_rejoined_seat failed before the fix: the rejoined row contained Direct { host: "old-endpoint", port: 7777 }. The deterministic pause is at the storage write. |
| Disposition | The handler now holds the client lifecycle gate through the metadata write. The regression passes with the new seat's metadata unset (#647). |
ARM-C011 — Reconnect can restore an old Direct endpoint¶
| Field | Record |
|---|---|
| State, severity | Fixed, medium |
| Player impact | After a network change, peers can receive the disconnected socket's old Direct host endpoint in a new SessionPlan. |
| Source and revision | src/reconnection.rs::register_disconnection_with_identity, src/server/reconnection_service.rs restore, and src/server/session_policy.rs host selection; reviewed at e615b0da. |
| Invariant | A new socket must advertise its own peer endpoint. The old disconnect snapshot kept PlayerInfo.connection_info, and reconnect restored it unchanged. |
| Confidence and reproduction | scripts/dev-loop.sh reconnect_preserves_join_time_but_clears_old_peer_endpoint failed before the fix: the restored row retained Direct { host: "old-network.example", port: 7777 }. The test also checks the saved join time and name. |
| Disposition | Disconnect registration now clears the old socket's peer metadata while retaining identity and replay state. The reconnect and duplicate-registration regressions pass. Existing Direct host plan tests verify that a host without an endpoint is re-elected or falls back to relay (#647). |
ARM-C012 — Old transport status can reach a rejoined seat¶
| Field | Record |
|---|---|
| State, severity | Fixed, medium |
| Player impact | Peers can receive a stale PeerTransportStatus after its sender leaves and rejoins. This can show an old direct or WebRTC path as live. |
| Source and revision | src/server/message_router.rs::TransportStatusFanOut::deliver, reviewed at 55a2b3a2. |
| Invariant | A status from one membership generation must not publish in a later generation. The old fan-out rechecked recipients but not the sender. |
| Confidence and reproduction | scripts/dev-loop.sh transport_status_delivery_does_not_block_concurrent_leave failed before the fix with old status reached the new seat after a paused delivery resumed following leave and rejoin. |
| Disposition | Fan-out now checks the sender's room membership generation at recipient queue commit, after any capacity wait. A canceled event does not increment the fan-out counter. The focused regression and transport-status tests pass (#647). |
ARM-C013 — A failed room rollback leaves a joinable unfinished room¶
| Field | Record |
|---|---|
| State, severity | Fixed for the in-memory backend, medium; broader #658 remains open |
| Player impact | If creator setup and deletion both fail, another player can join the unfinished room. A failed spectator-cap write then leaves that room unlimited. |
| Source and revision | src/server/room_service.rs unpublished admission rollback and src/database/mod.rs room lookup, reviewed at c85ee60b. |
| Invariant | A newly created room is invisible until its creator response commits. An abandoned room reserves its code until storage confirms deletion. |
| Confidence and reproduction | scripts/dev-loop.sh failed_cap_write_and_delete_keep_unpublished_room_closed failed before the fix: the next join received RoomJoined for the unfinished room. The green test checks a second server instance, spectator refusal, repeated repair failure, one deletion count, and code reuse after repair. Name-write and failed-read regressions cover sibling rollback paths. |
| Disposition | In-memory creation starts pending. The response builder publishes while holding the room code and mutation locks, before the queue commit; cancellation rolls back under both locks. Failed rollbacks are marked abandoned, and maintenance retries deletion by ID after checking that state again. The trait default now refuses server creation without a pending lifecycle and refuses direct protected creation without an atomic seal. Process-loss behavior and other #658 acceptance cases remain open. |
ARM-C014 — A creator setup panic leaves a pending room outside repair¶
| Field | Record |
|---|---|
| State, severity | Fixed, medium |
| Player impact | A storage panic after room insertion can reserve its code and room quota until process exit. The room remains invisible, so creators cannot use that code. |
| Source and revision | src/server/room_service.rs creator cap and name setup, reviewed at 49497a44. |
| Invariant | A failed creator setup must delete the pending room or mark it abandoned for repair before the creation lock is released. The outer panic supervisor lacks the room ID until setup returns. |
| Confidence and reproduction | scripts/dev-loop.sh creator_setup_panic_keeps_pending_room_repairable failed before the fix: injected cap-write panic left no abandoned room for cleanup. The green test covers cap and name panics followed by a delete failure. |
| Disposition | Creator setup catches a storage panic while it still owns the room ID, rolls back or marks the room abandoned, and returns a creation failure so the locks release. The focused regression checks the response, metrics, repair, and retry. Interrupted creation after insertion is covered by ARM-C015. |
ARM-C015 — An interrupted creator reserves a hidden room indefinitely¶
| Field | Record |
|---|---|
| State, severity | Fixed for the in-memory backend, medium |
| Player impact | If the creator stops after the atomic pending insert and before rollback, its hidden room consumes quota and reserves the requested code until shutdown. |
| Source and revision | src/database/mod.rs pending creation and src/server/maintenance.rs repair, reviewed at 630b5d3e. |
| Invariant | Repair may delete a Creating room only after its creator operation ends. A lost room-code lease alone does not prove the creator stopped. General room expiry must leave every unpublished room to the dedicated repair path. |
| Confidence and reproduction | interrupted_creation_releases_hidden_room_and_reserved_code failed before the fix because maintenance scanned only Abandoned rooms. The green test drops the creator token, lets its code lock expire, then checks deletion, quota, metrics, and code reuse. pending_room_repair_preserves_active_creator_after_code_lease_expires keeps the token alive past lock expiry and confirms publication succeeds. generic_room_cleanup_keeps_unpublished_rooms_for_repair covers both general reapers. |
| Disposition | The in-memory insert stores the original creator ID and a weak operation token under the same commit as the room, code, and pending marker. The response path holds the token through first publication. Repair takes the code lock and atomically rechecks the marker, owner liveness, and creator-only membership before deletion. A process restart drops all in-memory rooms and codes; durable adapters must provide their own process-loss recovery. |
ARM-C016 — A creator insert panic leaves lifecycle counters unbalanced¶
| Field | Finding |
|---|---|
| State, severity | Fixed for the in-memory backend, low; broader #658 remains open |
| Player impact | Dashboards undercount rooms and joined players when storage inserts a pending room and panics before the server resumes. Later repair can count deletion without a matching creation or departure. |
| Source and revision | src/database/mod.rs pending insertion and deletion; src/server/room_service.rs admission rollback, reviewed on this branch. |
| Invariant | A committed pending insert counts one room and creator before another suspension point. Removing that room counts the creator leaving exactly once, including direct deletion and repeated repair. |
| Confidence and reproduction | panic_after_pending_insert_balances_creation_metrics_once failed before the fix with rooms_created=0 after storage committed the pending room. It now covers failed then successful repair and all four counters. direct_pending_room_delete_balances_creator_metrics_once checks direct deletion and repeated leave accounting. |
| Disposition | The in-memory insert records creation and join while holding its commit guards. Rollback and deletion share one atomic accounting token for the departure. The trait compatibility default cannot provide the same guarantee for other backends; durable process-loss recovery remains in #658. |
ARM-C017 — Kick loses a reconnected seat during a second disconnect¶
| Field | Finding |
|---|---|
| State, severity | Fixed, high |
| Player impact | A kicked player can retain a fresh reconnection record and restore the seat after the authority receives PlayerKicked. |
| Source and revision | src/server/moderation.rs disconnected-target eviction, reviewed at 3ed1c337. |
| Invariant | A kick either removes the restored live seat while holding its current lifecycle gate or tombstones that room's pending record under the room event gate. A second disconnect cannot re-arm a restorable seat before removal, and the kick cannot revoke a newer credential for another room. |
| Confidence and reproduction | kick_cannot_leave_a_fresh_reconnect_record_after_target_disconnect failed before the fix with an untombstoned pending record. It pauses after kick validation, reconnects the target, then pauses after kick reads the restored route while the target disconnects. The green test verifies that disconnect cannot arm another record before removal and that no claimable record remains. kick_does_not_tombstone_a_new_pending_record_in_another_room checks a newer room B record against kick's stale room A validation. Existing active-kick and pending-seat tests remain green. |
| Disposition | Eviction rejects a stale target lifecycle guard after waiting, then rechecks the current lifecycle after its first tombstone. It holds a restored socket's lifecycle gate through removal, or rechecks and tombstones a pending record under the room gate if no socket exists. Both tombstone writes require a fresh same-room record check. |
ARM-C018 — Cyclic moderation waits on lifecycle locks¶
| Field | Finding |
|---|---|
| State, severity | Fixed defensive lock-order gap; reachable player impact unconfirmed |
| Potential impact | If live authorities have cyclic stale storage rows, simultaneous kick or ban requests can wait forever on each other's lifecycle locks. |
| Source and revision | src/server/moderation.rs::resolve_kick_style_target and evict_member_by_authority, reviewed at e4ae5526. |
| Invariant | Moderation operations must not hold lifecycle locks in a wait cycle. A late reconnect can create the target lifecycle after initial validation, so both acquisition sites need the same coordination. |
| Confidence and reproduction | A synthetic cross-room stale-row test, crossed_stale_seat_moderation_completes_without_lifecycle_deadlock, failed before the fix: both authority locks were held and the first kick timed out after one second. It covers kick/kick and kick/ban pairs; a three-way test covers longer cycles. The shipped in-memory failure path does not establish this exact crosswise state: leave_storage_error_preserves_membership_routing_and_reconnect_token keeps the route, disconnect_storage_error_forces_terminal_teardown_and_keeps_claim_reachable leaves a room-bound reconnect record, and disconnect_storage_error_retries_without_reconnection_support checks repair. No observed player incident is claimed. |
| Disposition | One moderation lifecycle gate serializes kick and ban while they may hold two lifecycle locks, including the late target reacquisition. It releases before the terminal result send. The synthetic regression and existing moderation cases check completion and preservation of unrelated live memberships. Other client-lifecycle users take one lifecycle lock per operation. |
ARM-C019 — Failed reconnect keeps an unrouted authority¶
| Field | Finding |
|---|---|
| State, severity | Fixed, high |
| Player impact | A failed reconnect can leave its unrouted player as room authority while storage removal is unavailable. Live members cannot take authority until detach repair succeeds. |
| Source and revision | src/server/reconnection_service.rs::rollback_claimed_reconnect, reviewed at 1be93b32. |
| Invariant | A reconnect that cannot deliver its baseline must release any authority it gained, even if its restored membership row cannot yet be removed. |
| Confidence and reproduction | cargo nextest run --lib -E 'test(failed_reconnect_rollback_does_not_leave_an_unrouted_authority)' failed before the fix: the room retained authority_player: Some(reconnecting) after baseline delivery and rollback removal failed. |
| Disposition | Rollback now clears authority when membership removal fails and keeps the failed detach queued. The regression checks that a live member can claim authority before repair, repair removes the row, and the original token can retry (#647). |
ARM-C020 — Stale terminal unroute deletes a newer route¶
| Field | Finding |
|---|---|
| State, severity | Fixed at the coordinator seam, medium |
| Potential player impact | If a stale room-A terminal unroute followed a move to room B, room B could stop receiving broadcasts. A move to the roomless lobby could lose directed responses. The reviewed shipped socket paths exclude this order. |
| Source and revision | src/server.rs::unroute_local_client_with_tail and src/server/connection_manager.rs::clear_room_assignment_with_tail, reviewed at b7e11dea. |
| Invariant | Refusal to clear a foreign room assignment must preserve the current room route or roomless delivery handle. The old coordinator swept every room route and removed the handle even when the assignment callback returned no terminal tail. |
| Confidence and reproduction | cargo nextest run --lib -E 'test(stale_terminal_unroute_preserves_new_room_route)' failed before the fix with left: Some([]), right: Some([player_id]) after room A to B. The same test covers A to roomless after the fix. |
Shipped-flow reachability at 8d47df25 |
Excluded for the reviewed WebSocket paths. leave_room_locked_operation is the only server caller of terminal unroute. Explicit leave holds its source ClientLifecycle gate through that call; disconnect and kick/ban hold the same player's gate when they call leave. Join and spectator admission also hold that gate, so they cannot publish another role first. Disconnect removes the old connection only after terminal unroute; reconnect refuses a target ID while that entry exists. Owned tasks retain the gate if their caller is cancelled. Missing-room maintenance uses the same gate, and shutdown uses quiet unregister without terminal unroute. The room A to B/lobby order in the coordinator regression is built with lower-level test hydration APIs, not a shipped socket request. This proof does not cover direct embedder use of public test-hydration or no-lifecycle unregister APIs. |
| Disposition | A terminal unroute with no tail now removes only the named room route and leaves the direct delivery handle until explicit unregister. stale_terminal_unroute_preserves_new_room_route and leave_from_old_socket_cannot_use_replacement_lifecycle passed at 8d47df25. The defensive coordinator fix stays in place; #685 can close as a documented shipped-flow exclusion. |
ARM-C021 — Removed socket leave departs a restored player¶
| Field | Finding |
|---|---|
| State, severity | Fixed at the server operation seam, high |
| Player impact | An old leave can remove a player's restored room seat and send a second departure event after reconnect. |
| Source and revision | src/server/room_service.rs::leave_room_owned, src/server/message_router.rs, and src/websocket/connection.rs, reviewed at 5b79ac98. |
| Invariant | An old socket's leave must not act on a lifecycle later installed under the same player ID. The old missing-lifecycle fallback could reach the restored seat. A frame already in the receive task can also resume after its send task unregisters and capture the new lifecycle by ID. |
| Confidence and reproduction | cargo nextest run --lib -E 'test(leave_from_removed_socket_cannot_depart_reconnected_lifecycle)' failed before the fix: the old leave paused after observing no lifecycle, reconnect restored the seat, and the leave removed it (left: None, right: Some(room_id)). leave_from_old_socket_cannot_use_replacement_lifecycle checks the stale physical socket Arc through the owned transaction and both router leave forms. These are server tests; the WebSocket ordering follows from its independent send and receive tasks and the router's await before dispatch. |
| Disposition | The missing-lifecycle branch returns. WebSocket dispatch now carries its socket lifecycle to the router and the owned leave, which locks that exact Arc and verifies it still owns the player ID. The green tests check assignment, durable membership, routing, departure count, peer and player messages, and a valid leave from the new socket. ARM-C020 records the shipped-flow exclusion for #685. #686 tracks other old-socket operations. |
ARM-C022 — Old socket moderates after authority reconnect¶
| Field | Finding |
|---|---|
| State, severity | Fixed for kick and ban, high |
| Player impact | A kick or ban from a removed socket can evict a peer after its authority reconnects on a new socket. |
| Source and revision | src/server/message_router.rs and src/server/moderation.rs::resolve_kick_style_target, reviewed at 330e25f7. |
| Invariant | A moderation request must use the physical socket lifecycle that sent it. The old kick and ban handlers awaited the moderation gate, then looked up a lifecycle by player ID; reconnect could replace that lifecycle during the wait. |
| Confidence and reproduction | cargo nextest run --lib -E 'test(old_socket_moderation_cannot_target_peers_after_authority_reconnect)' failed before the fix: a paused old request resumed after authority reconnect and removed the target. The test covers both kick and ban, then verifies that moderation from the restored socket still works. |
| Disposition | The router passes the source lifecycle through both handlers. The resolver locks that Arc and rejects it if it no longer owns the player ID. #686 remains open for other socket-bound operations. |
ARM-C023 — Old socket relays as a restored sender¶
| Field | Finding |
|---|---|
| State, severity | Fixed for text and binary relay, high |
| Player impact | Peers can receive game data from a removed socket stamped as a player's restored connection. |
| Source and revision | src/server/game_data.rs, src/server/connection_manager.rs::next_relay_stamp_in_room, and src/websocket/connection.rs, reviewed at 330e25f7. |
| Invariant | A relay frame may receive a sequence stamp only while its physical socket lifecycle owns the sender ID and room. The old text handler could resume after the router check, and the binary path bypassed that check. Both stamped the replacement lifecycle by player ID. |
| Confidence and reproduction | cargo nextest run --lib -E 'test(old_socket_game_data_cannot_relay_after_reconnect)' failed before the fix: peers received both old text and binary frames with the restored sender's epoch. The green test also verifies that the restored socket's own frames receive sequence 1 and 2. |
| Disposition | Both paths carry the source lifecycle and check it at entry and under the connection entry lock when assigning a relay stamp. The WebSocket receive task also stops frames from an already replaced socket before parsing. A frame that was in progress during replacement can still charge a byte budget before the final stamp rejects it; #686 tracks remaining accounting and response effects. |
ARM-C024 — Old socket changes restored authority's room policy¶
| Field | Finding |
|---|---|
| State, severity | Fixed for unban, authority transfer, room access, and code rotation; high |
| Player impact | A removed authority socket can lift a ban, transfer authority, seal a room, or rotate its join code after that authority reconnects on another socket. |
| Source and revision | src/server/message_router.rs and src/server/moderation.rs, reviewed at 2d104af8. |
| Invariant | Every authority operation must hold the lifecycle of the physical socket that sent it through its room mutation, rather than look up the current socket by player ID after router dispatch. |
| Confidence and reproduction | cargo nextest run --lib -E 'test(old_socket_cannot_change_room_authority_or_access_after_reconnect)' failed before the fix: an old UnbanPlayer frame crossed the router check, the authority reconnected, and the old frame removed a seeded ban (left: false, right: true). The green test covers all four operations, unchanged storage and recipient queues for old frames, and successful operations from the restored socket. |
| Disposition | The router passes its source lifecycle into each operation. Each handler locks that exact lifecycle and rejects it if the player ID now maps to another socket. #686 remains open for other dispatch paths and relay accounting. |
ARM-C025 — Old socket changes restored state through direct message handlers¶
| Field | Finding |
|---|---|
| State, severity | Fixed for authority requests, readiness, game start, peer metadata, Signal, TransportStatus, and application Ping; high |
| Player impact | A frame paused after the router's connection check could resume after reconnect and use the restored player's current lifecycle, changing room state or sending peer-visible traffic. An old Ping could also charge the new socket's reply budget. |
| Source and revision | src/server/message_router.rs, src/server/authority.rs, src/server/ready_state.rs, src/server/game_data.rs, src/server/signaling.rs, and src/server/heartbeat.rs, reviewed at da3b10f8. |
| Invariant | Every socket-originated direct handler must lock the lifecycle of the physical socket that sent the frame and confirm it still owns the player ID before acting. |
| Confidence and reproduction | old_socket_authority_request_cannot_release_restored_authority failed before the fix: the old authority frame released the restored player's authority (left: None, right: Some(...)). Green paused-dispatch regressions cover authority, readiness, game start, metadata, Signal, TransportStatus, and Ping, plus valid frames from each replacement socket. |
| Disposition | The router carries its source lifecycle to all seven direct handlers. Each locks that exact lifecycle and checks the current mapping before mutation, fan-out, or budget charge. #686 remains open for admission, transport control frames, and pre-stamp relay accounting. |
ARM-C026 — Old spectator leave replies to a restored socket¶
| Field | Finding |
|---|---|
| State, severity | Fixed for plain and correlated spectator leave, medium |
| Player impact | An old LeaveSpectator request can send NotASpectator to the restored player's socket. Its room-operation form can also charge the restored socket's error budget during capability refusal. |
| Source and revision | src/server/message_router.rs, src/server/spectator_handlers.rs, and src/server/spectator_service.rs, reviewed at 38e5ccb5. |
| Invariant | A spectator leave and its reply belong to the physical source socket. A stale request cannot use a replacement socket's lifecycle or reply budget. |
| Confidence and reproduction | scripts/dev-loop.sh old_socket_spectator_leave_cannot_reply_to_restored_player failed before the fix: an old leave sent a failure to the restored socket. The test covers plain and correlated forms after a paused dispatch. |
| Disposition | The handler and detach service carry the source lifecycle and check ownership before detach and reply. The router checks ownership while it reads room-operation capability and sends a capability failure. The correlated spectator test covers valid leave and rejects a mismatched source lifecycle. #686 remains open for other admission, transport control, and relay accounting. |
ARM-C027 — Old socket joins or reconnects under a restored identity¶
| Field | Finding |
|---|---|
| State, severity | Fixed for room join, spectator join, and connection-task reconnect; medium |
| Player impact | An old socket frame can resume after a replacement reconnects and send a room-join, spectator-join, or reconnect failure to that replacement. It can also enter admission using the replacement's lifecycle. |
| Source and revision | src/server/room_service.rs, src/server/spectator_handlers.rs, src/server/spectator_service.rs, src/server/reconnection_service.rs, and src/websocket/connection.rs, reviewed at 744a2080. |
| Invariant | Each socket-originated admission transaction uses the physical source socket's lifecycle through its first state change and any failure reply. A replaced socket cannot adopt a current player ID. |
| Confidence and reproduction | old_socket_join_cannot_reply_to_restored_player, old_socket_spectator_join_cannot_reply_to_restored_player, and old_socket_reconnect_cannot_reply_to_restored_player each failed before the fix with an old failure delivered to the replacement. The tests cover plain and correlated forms. |
| Disposition | The router and WebSocket receive task pass the source lifecycle into owned join and reconnect transactions. Spectator join checks it before drain refusal and after a failed transaction. Positive room and spectator joins and correlated reconnect remain covered. #686 remains open for transport control, relay accounting, and other lifecycle capture points. |
ARM-C028 — Old control and invalid frames affect a restored socket¶
| Field | Finding |
|---|---|
| State, severity | Fixed for WebSocket receive control and refusal paths, medium |
| Player impact | An old socket can refresh a restored player's idle clock or send it an error and charge its reply budget after reconnect. |
| Source and revision | src/websocket/connection.rs receive loop, src/server/heartbeat.rs, src/server/game_data.rs, and src/server/messaging.rs, reviewed at c0b3c8b0. |
| Invariant | A socket-originated control frame or refusal can affect only the physical socket that sent it. The receive loop's first ownership check alone does not protect later awaited work. |
| Confidence and reproduction | scripts/dev-loop.sh old_socket_receive_frame_cannot_affect_restored_player failed with the old receive calls: malformed text reached the replacement, and Ping changed its heartbeat counter from 1 to 2. The green socket test covers malformed text, Ping, Pong, binary format refusal, and repeated Authenticate. A one-reply budget proves stale frames consume no replacement reply; a current refusal still succeeds. The helper regression also checks the idle clock and current activity. |
| Disposition | Ping/Pong liveness, receive-loop parse/size/encoding refusals, and game-data refusals now lock and verify the source lifecycle through their effects. Authenticate processing holds the source gate while it reads and changes connection policy. #686 remains open for relay accounting before stamping and other lifecycle capture points. |
ARM-C029 — Old relay charges a restored player's byte budget¶
| Field | Finding |
|---|---|
| State, severity | Fixed for relay budget admission, medium |
| Player impact | An old socket relay suspended at a budget wait could charge the restored player's sender and room byte windows for a frame the lifecycle-guarded stamp then rejected — a phantom charge that throttles the replacement's own game data. |
| Source and revision | src/server/game_data.rs, src/coordination/mod.rs, and src/server.rs, reviewed at 409482e5. |
| Invariant | Relay budget admission, stamp allocation, and enqueue complete under the sender's source lifecycle gate; the gate releases before any backpressured fan-out completion, so a reconnect never waits on queue drains. Charged bytes and room delivery agree for every admitted frame. |
| Confidence and reproduction | old_relay_budget_admission_serializes_restored_player_reconnect failed with the gate disabled: the reconnect rekeyed while the old relay sat paused at each budget wait. The green table covers text and binary lanes at both budget waits, proves the rekey waits for admission, and checks the charged-byte count against the delivered frame plus the replacement's unthrottled follow-up relay. |
| Disposition | Both relay lanes hold the source gate across the budget waits and the coordinator start. A new enqueue_relay_broadcast_after_contention seam splits the coordinator contention fallback so its routing waits stay under the gate and only the drain awaits outside it. Budget rejections reply after the gate releases. #686 remains open for the lifecycle-capture-point sweep. |
ARM-C030 — MessagePack conversion decode had no wire-level depth contract¶
| Field | Record |
|---|---|
| State, severity | Hardened, low (defense-in-depth; no production abort reproduced) |
| Player impact | Before this change, the conversion path's recursion budget was an implementation detail of one dependency version: rmp-serde 1.3's internal 1024-level guard. A 60 KB sub-cap fixarray chain requests ~60k levels, so an upgrade or replacement dropping that internal guard would have turned a cross-format relay into a stack-overflow process abort. Payloads nested 129-1023 levels, which JSON game data could never reach at the same boundary, converted successfully; they now refuse. |
| Source and revision | src/websocket/sending.rs::decode_binary_to_json decoded attacker-supplied MessagePack into a recursive serde_json::Value with rmp_serde::from_slice, reviewed at 48bdc7fb. The frame-size cap bounds bytes, not depth: one fixarray marker byte adds one nesting level. |
| Invariant | The MessagePack nesting the server decodes must sit under an explicit wire contract (128 levels, serde_json's own JSON limit), enforced before any recursive decode, not on the decoder's internal recursion counter. |
| Confidence and reproduction | scripts/dev-loop.sh over_deep_message_pack_is_refused_before_decoder_recursion runs the production decode on a 256 KiB stack: before the guard it aborted (fatal runtime error: stack overflow, SIGABRT), with the guard it returns the refusal (constant stack). On production 2 MB tokio worker stacks, guardless rmp-serde 1.3.1 stops at its internal 1024-level limit with a clean decoder error, so the abort class was latent, not live; the probe pins the guard against regressions on every platform. |
| Disposition | An iterative depth scanner (msgpack_depth_within, MSGPACK_MAX_NESTING_DEPTH = 128) walks the structure with an explicit stack before any recursive decode and refuses deeper trees as an undeliverable conversion, reusing the existing exact report and advisory accounting. The same scan guards the token-bound binary envelope (parse_binary_message). Malformed input stays a decoder error; the scanner only answers depth. The walk is allocation-free: its fixed-size sibling stack's capacity is the enforced limit itself, pinned by the #558 mixed-source allocation ceiling in relay_serialization_allocations (the first scanner draft cost one Vec allocation per relay and failed that ceiling on the per-PR lane). Differential review: a spec-faithful reference parser agreed with the scanner on 200k random well-formed payloads at limits {1, 2, 3, 4, 8, 128}, exhaustive 1- and 2-byte marker spaces, and all truncations; 100k mutation fuzz found no scanner-refused-but-decoder-accepted case. Pins: depth_scanner_matches_the_limit_boundary, depth_scanner_counts_map_entries_as_two_slots, message_pack_depth_limit_is_exact, depth_scanner_accepts_shallow_wires_and_skips_payload_bytes, depth_scanner_is_conservative_on_malformed_input. |
ARM-C031 — The rejected app-ID warning echoed the raw client-supplied ID¶
| Field | Record |
|---|---|
| State, severity | Fixed, low (log forgery; once per rejected connection) |
| Player impact | None on the wire: the rejection, error code, and close are unchanged. Operator impact: a client could forge or distort operator-facing log lines precisely when its app ID was rejected — the Public app ID rejected warning printed the raw ID with a Display field in the arm where the log-safety gate had just refused it for control characters (newlines, ANSI escapes) or length. |
| Source and revision | src/websocket/connection.rs authentication error arm, reviewed at dd37153b. The gate (app_id_is_log_safe) rejects unsafe IDs at resolve time; the Err arm then logged %app_id unescaped. |
| Invariant | A field the log-safety gate has not vetted must never reach a log line through a Display (%) field; client-chosen text in anomalous-path warnings is Debug-escaped. |
| Confidence and reproduction | Direct code reading: the Err arm is reached by every resolve failure, each carrying the unvetted client-supplied ID — the gate-rejection InvalidAppId variant (which also serves unknown IDs), RateLimitExceeded, and the reserved app-status variants. The sibling anomalous-path warnings (message_router.rs, room_service.rs spans) already document and apply the Debug-escape rule for exactly this hazard. |
| Disposition | The field is now ?app_id (Debug-escaped) with the rule restated at the site. Sweep: every remaining %app_id log site is gate-vetted (inside Ok(info) arms) or a typed UUID; the wire-level rejection contract stays pinned by test_unloggable_app_id_fails_authentication_with_invalid_app_id. |
ARM-C032 — The undeliverable-relay warning fired per frame per recipient¶
| Field | Record |
|---|---|
| State, severity | Fixed, low (log flood; bounded harm, no delivery impact) |
| Player impact | None on delivery: reports, advisories, and drop counters are unchanged. Operator impact: one sender relaying an encoding a recipient cannot convert (e.g. rkyv into a JSON-only room) produced one tracing::warn! per frame per recipient with no throttle, so a single mismatched sender into a large room could flood the log sink (CPU/disk pressure) while the in-band advisory beside it was already limited to one per sender per second. |
| Source and revision | src/websocket/sending.rs::notify_on_undeliverable, reviewed at dd37153b. |
| Invariant | A per-event log on a hot path must share the rate limit of the response it describes; per-event accounting belongs to counters, not to unbounded log emission. |
| Confidence and reproduction | Direct code reading against the advisory limiter (unsupported_notice, one notice per sender per second with a suppressed count): every undeliverable conversion passed the unthrottled warn before any cadence check. |
| Disposition | The warning now rides the advisory cadence and carries the suppressed count; the fail-closed missing-metadata error log is unchanged apart from gaining the same encoding/reason fields the old warn carried. The per-event totals stay observable through the delivery ledgers and drop metrics pinned in the mixed-encoding and volatile-loss reviews. Per-sender cadence holds under any sender count; as with the pre-existing advisory limiter, total log volume scales with distinct offending senders, bounded by concurrent connections. |
These findings cover room-code rotation, player names, transport status, and spectator, reconnect, room-creation drain, and terminal routing seams, plus the two log-bound hardenings above. The rest of the C1 room and storage rows remain unreviewed.
C1 admission-limit review (2026-09-28)¶
At 46f840ab, reviewed the in-memory room seat check and the server and
application room-count gates. concurrent_joins_for_last_seat_admit_one_player
parks the first seat write while a second player joins the same full-boundary
room. It checks one admission, one ROOM_FULL refusal, the stored roster, and
both routes. Existing barrier checks
server_room_cap_is_atomic_across_games and
application_room_cap_is_atomic_across_games_and_independent_between_apps
cover concurrent creation in different games at the server and application
limits. join_racing_room_deletion_reports_room_not_found now checks both
ordinary and join_only admission when a room vanishes before the membership
write. Both receive ROOM_NOT_FOUND. No violation was reproduced in these
paths. Other storage adapters, leave/disconnect races, and the remaining C1
cases are unreviewed.
C1 leave/disconnect ordering review (2026-09-28)¶
At 56803cf0, checked the socket lifecycle gate where an explicit leave
waits behind disconnect after the reconnect claim is armed, and where
disconnect waits behind a leave stalled on peer delivery. The regressions
leave_queued_during_disconnect_cannot_consume_reconnect_or_repeat_departure
and disconnect_queued_during_leave_does_not_arm_a_reconnect confirm one
absent durable row, one peer PlayerLeft, one departure count, and the expected
token outcome for each order. Neither interleaving reproduced a defect. Other
leave/disconnect races remain unreviewed.
C1 reconnect and activity-reaper ordering review (2026-09-28)¶
At 24e1dafb, checked the case where activity cleanup selects an expired
transient socket before reconnect moves that socket to the restored player ID.
stale_reaper_snapshot_cannot_close_a_restored_reconnect advances paused time,
captures the cleanup candidate, reconnects, then applies the stale candidate
through the farewell and fallback close paths. The test checks the restored
room seat, route, peer event, consumed token, and usable socket. No violation
was reproduced. The opposite order, where cleanup pins a close before the
reconnect claim, is covered by
reconnect_on_a_reaper_pinned_socket_is_refused_and_preserves_the_token.
Other stale-socket cleanup paths remain unreviewed.
C1 reconnect claim expiry during restore (2026-09-28)¶
Reviewed the claimed reconnect while its membership write is paused across
the monotonic admission deadline. The unit regression
reconnect_claim_survives_expiry_during_membership_restore runs the server's
expired-record cleanup after the deadline, then releases the write. Cleanup
removes an expired unclaimed sibling but retains the claimed record and room
protection; the original seat, route, peer
event, and directed baseline complete, and the one-time token is consumed.
No violation was reproduced. The manager's existing
expired_but_claimed_reconnection_survives_every_expiry_surface test covers
claim release after expiry and subsequent cleanup. Other failed restore and
retry paths remain unreviewed.
C1 lifecycle-capture-point sweep (2026-09-30)¶
Closes the last open row of
#686
at 8191ed60. The sweep reviewed every site that captures or validates a
ClientLifecycle before one or more await points and then acts, across all 17
files that mention the type: server.rs, server/{authority,connection_manager,
game_data,heartbeat,maintenance,message_router,messaging,moderation,ready_state,
reconnection_service,room_service,signaling,spectator_handlers,
spectator_service}.rs, and websocket/{connection,handler}.rs. The rekey
mutates the shared lifecycle only from the gate-holding reconnect transaction
(reconnection_service.rs:678 guards reassign_connection at :1313;
set_player_id has no other caller), so a held gate pins the identity. A site
is safe when it re-validates player_id() plus lifecycle_matches under the
gate after its last pre-act await, or when its act is keyed to state a rekey
removes.
Every lifecycle-resolving handler follows the fenced shape: authority, ready
state, start game, transport ping, signal dispatch, transport status, join,
leave, spectator join/detach, moderation (kick/ban/unban/access/transfer/
rotation, with guards carried through ModerationTarget), relay text/binary
admission (the stamp adds a second ptr_eq fence at the final touchpoint),
authenticate and receive-loop refusals, assign_client_to_room,
cleanup_client_in_missing_room, and the reconnect transaction itself.
Documented exclusions, each verified against the code:
- Ungated keyed-liveness lanes (
maybe_update_last_seen, the router's pre-dispatch activity writes, the roomless binary branch): the stale id's map entry is gone after a rekey and unknown ids never take a throttle stamp, so the write lands nowhere or only on the retired identity; the router re-checks the source after the await before dispatching. - Socket-keyed acts (probe state, farewell enqueues, close pins from the socket's own I/O tasks): the signal belongs to the physical socket and a rekey cannot redirect it.
- Post-release replies (moderation terminal results, relay budget refusals, spectator failure replies): keyed to an id whose route the rekey removes, or re-locked and re-validated before sending.
- Test-only entries:
handle_reconnectwith no lifecycle is unreachable from production dispatch; the socket path always forwards the source lifecycle.
No violation was reproduced. One defense-in-depth residual was found and
fixed: charge_error_reply pinned the 4006 close by map key after the
farewell await, so a rekey landing inside that await skipped the pin, and the
one-shot report_exhaustion never retried it until the window rolled over.
The window was unreachable (every charge site holds the charging player's own
lifecycle gate, which the rekey needs), but a future ungated charge site would
have reintroduced it. The close now pins the per-socket close signal captured
under the charge guard, so it follows a concurrent identity swap
(#697,
fixed by PR
#699,
regression
error_reply_exhaustion_pins_the_close_through_a_rekey_inside_the_farewell_await).
The same capture-under-fence pin hardened the authority kick's one-shot
4007 close (evict_member_by_authority), the last same-class residual
found by the fix review.
With this sweep, #686
is closed: each path has a deterministic regression from PRs #687-#696 or a
documented exclusion here.
C1 reconnect failed-restore and retry review (2026-09-30)¶
At bdf8bddb, reviewed every post-claim rejection in handle_reconnect_owned
(src/server/reconnection_service.rs): the pre-restore refusals (room full,
missing seat record, name conflict), the kicked, already-connected, banned,
and wrong-application-id rechecks inside the claimed window, the
membership-write storage fault and its room-missing reclassification, the
stale-endpoint clear failure, the drain refusal, the post-restore authority
grant, the room-readback faults, and the reassignment and
baseline-publication failures. The rejection
contract holds on every reviewed path: reject_claimed_reconnect rolls back
exactly the state the attempt made durable (restored membership row, granted
authority, taken-over detach retry with its application-claim provenance),
releases the claim unspent, and replies with the classified code. The
Option<Option<_>> detach-requeue arithmetic re-queues inherited provenance
unchanged when the rollback removal fails (promoting a bare inheritance to
an owed bare retry), and when removal succeeds retains only
ownership-carrying provenance — a bare inheritance then means nothing is
left to repair.
Evidence added:
reconnect_membership_restore_storage_error_releases_claim_for_retry— a transient fault on the membership-restore write (new one-shotfail_next_add_player_to_room_for_testinjection) repliesInternalError, leaves the roster untouched, releases the claim, and the same token reconnects once storage recovers. Red-proofed by disabling the storage-error rollback.reconnect_authority_grant_storage_failure_completes_degraded_and_recovers— an authority-grant fault (new one-shotfail_next_request_room_authority_for_testinjection) after membership restored completes the reconnect degraded: live member, vacant role, no presented authority, and the member's next reconnect re-runs the grant. Red-proofed by making the grant failure reject the attempt. Disposition: best-effort continuation is correct — the membership row is already durable when the grant runs, so rejecting would owe a removal to the storage that just failed; recovery is the client-driven reconnect any live was-authority member can perform.- The room-deleted reclassification, room-full, name-conflict, drain,
reassignment-failure, and baseline-failure rejections were already pinned
by
reconnect_room_deleted_during_restore_is_classified_room_not_found,reconnect_room_full_failure_releases_claim_for_retry,reconnect_name_taken_by_new_member_rejects_without_spending_token,reconnect_during_shutdown_drain_is_rejected_with_server_draining,reconnect_reassign_failure_rolls_back_membership_and_releases_claim, andreconnect_baseline_delivery_failure_rolls_back_and_releases_claim_for_retry. - The bare-detach (
Noneprovenance) requeue branch is covered by derivation plus the ownership-provenance siblingrejected_reconnect_requeues_the_ownership_rollback_it_inherited; no dedicatedNone-provenance pin was added. - The remaining gates carry recorded evidence or an explicit exclusion: the
banned refusal is pinned end to end through
handle_reconnect(ErrorCode::Banned,moderation_tests.rsban-reconnect regression), the wrong-application-id refusal byapp_bound_room_owner_gates_seated_spectator_and_reconnect_admission(wrong app →RoomNotFound), the claim-level kicked refusal by the moderation unit pin (claim_reconnection→Err(Kicked)), and the pre-claim already-connected refusal at the dispatch boundary (reconnect_during_teardown_preserves_token_for_retryand the H6 duplicate-claim suite). The stale-endpoint clear failure, the pre- and post-restore room-readback faults, and the kicked/already-connected rechecks inside the claimed window are covered by derivation only: each exits through the samereject_claimed_reconnectseam whose storage-fault and rollback branches are red-proofed here, injects no state a later check trusts, and releases the claim through the same guard. No dedicated pins were added for them.
No violation was reproduced. Every post-claim rejection of the reconnect restore has a recorded disposition: each gate is pinned end to end or carries an explicit derivation exclusion in this section.
C1 token rotation boundary review (2026-09-30)¶
At 562a63fb, reviewed the rotation boundaries of the reconnect transaction
(src/server/reconnection_service.rs) against the concurrent-claim guards
(src/reconnection.rs). The invariant chain: claim_reconnection checks and
sets the claim under one write-lock hold with no awaits, so a second claimant
during the transaction gets AlreadyInProgress (pinned by
test_reconnection_claim_is_single_use_under_concurrency, the H6
duplicate-claim e2e, and the fuzz model's no-double-claim invariant). The
disconnect registration consumes the pre-issued join token, so no credential
exists for the identity between disconnect and rotation. Rotation mints the
fresh token as the last step of the Reconnected baseline builder, inside the
coordinator's registration critical section; completion removes the claimed
record only after that baseline is queued, so the token is never consumed
without a delivered baseline. The new token is unclaimable until the next
genuine disconnect arms it, and a retry presenting the consumed token after
a completed reconnect is refused — NoRecord while the consumed record
stays removed, and by token comparison once the next disconnect has armed
the fresh token.
Failure unwinding after rotation: a rejection once reassignment has occurred
runs reject_after_reassigned_reconnect_failure, which discards the freshly
rotated pre-issued token before restoring the transient identity and rolling
back, so a player who was never restored never holds an armed credential. The
panic supervisor repeats the same unwinding for the reassigned-but-not-committed
phase (ReconnectAfterReassignment regression) and, post-commit, completes the
claim and keeps the rotated token (ReconnectAfterTerminal regression pins the
consumed old token). ReconnectionClaimGuard::drop deliberately releases
nothing; only the supervisor's phase-aware completion or rollback may mutate
the claim.
Evidence added (extends
reconnect_baseline_delivery_failure_rolls_back_and_releases_claim_for_retry):
- A delivered retry must surface a rotated token on the wire (
Reconnectedpayload token differs from the consumed one) and re-arm the pre-issued map for the next disconnect. Previously only the consumed old token was pinned; the fresh-credential side of the rotation was not. - A failed baseline delivery leaves no pre-issued token for the unrestored identity.
- Boundary record: in that test's queue-full scenario the coordinator refuses at initial-slot reservation, before the baseline builder runs, so the rotation has not happened yet and the reject-path discard is a no-op there. The post-rotation failure windows (builder fault, commit channel-close, drain flip between builder and commit) share this discard line and the same unwinding as the pinned reassigned-failure path.
- Post-rotation drain-flip pin (issue #707, 2026-09-30): a new one-shot
get_room_playerspause seam parks the reconnect transaction inside the baseline builder, before the builder's final rotation step. Flipping the shutdown drain while parked makes the post-builder commit gate refuse withDeliveryOutcome::Canceled— the exact drain-flip window — so the reject path must discard the freshly minted token.reconnect_drain_flip_after_baseline_rotation_discards_the_fresh_tokenpins the full end state:ReconnectionFailed(ServerDraining)on the wire, no pre-issued token for the unrestored identity, restored transient identity, claim released for retry, and rolled-back membership. Red-proofed by removing the discard line: the fresh token survives armed and the test fails on the pre-issued-token assertion. The builder-fault and channel-close windows keep derivation-only status for their own arrival paths, but they exit through this same now-pinned discard line.
No violation was reproduced. The rotation ordering, concurrent-claim refusals, both unwinding phases, and the post-rotation reject-path discard (through the drain-flip window) carry pinned evidence.
C1 allowlist and key reload boundary review (2026-10-01)¶
At a6b6214e, reviewed the SIGHUP reload seam end to end: the glue
reload_allowed_apps_from_config (src/main.rs), the allowlist swap
AppIdAllowlist::reload (src/auth/middleware.rs), the key and posture swap
EnhancedGameServer::install_connect_token_key (src/server.rs), the
handshake consumption point (src/websocket/connection.rs Authenticate),
and the configuration sources (config::load folds
public_key_path into public_key at load, so the reload reads the same
sources as startup and an unreadable key file is a load error, never a
keyless server).
The invariant chain: a config that fails to load or fails
validate_config_security applies nothing — the gate runs before both swaps,
and the two swaps are synchronous calls with no await between them, so a
partially applied reload is unreachable on the SIGHUP path. A valid SIGHUP
applies the allowlist swap and the key swap together: revoked apps stop
resolving for fresh handshakes, added apps resolve, and live connections keep
their resolved context by design (live revocation stays a restart-level
action). The install order is posture-before-key: arming briefly publishes
required-with-old-key (token-less handshakes refused — fail-closed), and the
reverse order is the only one that could open a required-but-unenforced
window. A presented token with no key installed is refused NoKeyConfigured,
so removing the key refuses closed, and every mixed state refuses closed.
install_connect_token_key re-parses the
key, but validate_config_security parses the same key first, so a
corrupt-key SIGHUP is rejected before either swap and cannot regress below
the gate.
Evidence added (binary connect_token_reload_tests, through the SIGHUP
glue; the allowlist state is observed with a public diff probe — re-applying
a set reports an empty diff exactly when it is the running set):
sighup_security_invalid_config_keeps_the_running_allowlist_and_key: duplicate app IDs (carrying an armed posture and a fresh key), and arequire_connect_token=trueentry with the key block removed, each keep the running allowlist, the running key (old tokens verify; the rejected config's key does not install), and the running posture. Red-proofed by disabling the validation gate: with the gate gone, the duplicates config installs its key and arms the posture, and the test fails on key retention. The duplicates variant is additionally defended in depth by the swap's own entry validation (AppIdAllowlist::reloadrejects it again); the no-key variant isolates the glue gate alone.sighup_reload_applies_allowlist_and_key_swaps_together: one valid SIGHUP revokes app-a, adds app-b, and rotates the key; the swapped set is live, fresh-key tokens verify, and old-key tokens stop verifying in the same pass. Red-proofed by disabling the allowlist swap: the probe fails on the swapped-set assertion.
No violation was reproduced. The reload boundary — key/allowlist swap order and invalid reload — carries pinned evidence through the SIGHUP glue.
C1 identity-slice completion review (2026-10-01)¶
At 81cc153a, closed the three remaining identity and membership case
families of #647:
spectator transitions, kick/ban races, and application isolation.
Spectator transitions. Every role-mutating path serializes on the
player's ClientLifecycle gate: a voluntary leave locks it through
detach_expected and validates ownership after the wait; the disconnect
teardown holds it across unregister_client_locked, whose spectator
detach_if runs inside that hold; join, moderation, and reconnect admission
carry their source lifecycle (ARM-C021/C026/C027). The owned detach
re-validates the player→room mapping under the room mutation gate (the
issue-241 TOCTOU fence), is idempotent on an absent entry, and the prune
path refuses a moved or gone session through expected_room. Kick and ban
refuse spectator targets by contract (KickTargetNotFound — the authority
documentation scopes both operations to seated players; no Kicked
spectator reason exists). The one unpinned interleaving — voluntary leave
racing the disconnect teardown — is now pinned in both orders
(spectator_leave_wins_disconnect_race_with_exactly_one_voluntary_detach,
spectator_disconnect_wins_race_and_the_late_leave_detaches_nothing):
exactly one detach wins, room events carry the winner's reason, the loser
is inert (no second event, StorageError without state change), and the
roster, local role, and retry backlog end clean. Red-proofed by disabling
both serialization fences (the detach_expected lifecycle fence and the
detach_owned room-gate re-validation): the leave-wins order publishes a
phantom second spectator event, and the disconnect-wins order lets the
losing leave report success; the tests fail on those assertions. A losing
leave whose pre-gate is_spectating probe predates a completed detach
replies StorageError to a socket the disconnect already removed — the
reply has no route and no state changes, so it is inert by derivation.
Kick/ban races. The ban write serializes on the room mutation gate
ahead of the eviction, and the reconnect claim re-reads the room
gate-fresh and refuses a recorded ban (issue #525 pins both orders). A kick
of a disconnected target tombstones under the room gate and fences the
second-disconnect re-arm (ARM-C017); moderation serializes through one
lifecycle gate against lock cycles (ARM-C018); old-socket kick, ban,
unban, transfer, access, and rotation are fenced at the source lifecycle
(ARM-C022/C024). A kick that races a voluntary leave waits on the target's
lifecycle gate, then re-reads storage truth under it
(resolve_kick_style_target revalidation): the seat is either still held
and evicted, or already gone and refused KickTargetNotFound — no
intermediate state is observable. Covered by derivation from the fenced
validation ordering.
Application isolation. The app-owner gate covers every admission
perimeter in both allowlist and open modes: fresh JoinRoom
(non-enumerating RoomNotFound), JoinAsSpectator, and reconnect
(app_bound_room_owner_gates_seated_spectator_and_reconnect_admission,
including persistence-based authorization after a cache loss and
token-preserving wrong-app refusals), plus open-policy scoping
(open_policy_rooms_are_scoped_to_their_application), atomic application
room-cap claims, and per-app player caps. Moderation is room-scoped by
resolution (the target must be a member of the authority's own room), so
no cross-app moderation path exists. No violation was reproduced.
With this review, every case family in the identity and membership slice has a recorded disposition: pinned end to end, or derived from a fenced ordering with named evidence. The slice is complete.
C1 gameplay-transitions review (2026-10-01)¶
At 9e553041 (main after #712), reviewed the gameplay-transitions case
families of #647:
ready-state invalidation on membership change, start/leave races, authority
election and loss, host/direct readiness, v2/v3 negotiation, transport
capability intersections, stale transport reports, reconnects with changed
encoding or capabilities, and the publication order of room snapshots,
session plans, and gameplay events. Every family already carries pinned or
derived evidence; the review found one unrecorded fail-closed coupling and
pinned it.
Start authorization trusts room assignment. The coordinator's
handle_start_game_with_publication authorizes "the designated authority,
otherwise any member" but never re-checks that the sender is a seated member
under the room mutation gate; the handler-level get_client_room lookup is
the only membership evidence ("the sender is already known to be in this
room"). The shipped dispatch paths keep that trust sound: a spectator's
connection is never assigned a room (join_owned requires
get_client_room(..).is_none() and never assigns one), so a spectator's
StartGame refuses NOT_IN_ROOM before the coordinator; a departing
player's leave serializes on the sender's own ClientLifecycle gate, which
the start handler holds from dispatch through the coordinator call, so no
leave can interleave between the room lookup and the gate acquisition. The
soundness is one refactor away from breaking: giving spectator connections a
room assignment (for example to deliver snapshots or broadcasts — the gap
spectator-mode.md records as a known limitation) would let a spectator
finalize any authority-less room whose players are all ready. The coupling
is now pinned (spectator_start_game_cannot_finalize_the_lobby):
dispatching StartGame from a joined spectator through the real router
against a fully ready authority-less lobby requires the NOT_IN_ROOM
refusal with the lobby still open and the ready set intact. Red-proofed by
performing that exact refactor in a probe (assigning the spectator's
connection to the room): the start then finalizes the lobby and the test
fails on the refusal assertion. spectator-mode.md's "What Spectators
Cannot Do" now lists StartGame explicitly.
Case-family dispositions (reviewed at 9e553041; no new violations):
- Ready-state invalidation on membership change. Joins deliberately
break a cached
all_readywithout a corrective broadcast; the authoritativeStartGamegate recomputes readiness over routed membership under the room gate (#447 F1 decision b,join_breaks_cached_all_ready_without_a_corrective_broadcast). Rejoining does not restore stale readiness; a reconnect into a finalized room restores that member's recorded readiness; room snapshots read the coordinator ready set throughsnapshot_ready_players(finalized rooms report their final readiness); departures prune the departing id directly and ready toggles prune non-members. - Start/leave races. Finalization commits through
commit_room_messages_if_membersplus thefinalize_room_gameCAS on (members, authority, lobby state): a member leaving before the commit yieldsRoutingChanged(bounded retries, then the terminal-boundary fallback), and the storage-level ghost-row case is the documentedSnapshotChangeddeferral (session-193, #396). The exact-membership snapshot gate is pinned (handle_player_ready_finalizes_with_member_snapshot_matching_game_starting_peers,start_game_builder_gates_each_plan_by_that_exact_members_version), and old-socket lobby frames are lifecycle-refused (old_socket_lobby_frames_cannot_toggle_ready_or_start_game). - Authority election and loss.
request_room_authorityis membership-gated (NotAMember) and refuses a held designation (AlreadyHeld); a departure that actually removed the authority clears the designation and everyis_authorityflag (no auto-reassign, per protocol); reconnect authority restore is live and replay-visible, never overrides a successor, degrades explicitly on storage failure, and rolls back without leaving an unrouted authority (pins insignaling_tests). - Host/direct readiness. Host+Direct plans carry empty ICE, and a host
without a validated endpoint falls back to the relay floor
(
emit_host_direct_room_carries_empty_ice_even_with_turn_enabled,emit_host_direct_room_without_endpoint_falls_back_to_relay). - V2/v3 negotiation and capability intersections. The selection ladder
is table-pinned rung by rung including every downgrade
(
selection_table_resolves_each_rung_and_downgrade); the finalized seat-fill gate demands v3 plus both sticky axes (seat_fill_predicate_tracks_version_and_both_sticky_axes); signal transport gates are v3+WebRTC on both ends (signal_sender_must_be_v3_even_if_webrtc_transport_is_present,signal_to_v2_peer_reports_target_not_found,signal_to_v3_relay_only_peer_reports_target_not_found). - Stale transport reports. ARM-C012's post-leave/rejoin status fix is pinned by the transport-status cohort (dedup without re-fan-out, flap fan-out per transition, budget bounding, slow-peer sharing, non-v3 ignored).
- Reconnect with changed encoding or capabilities. A downgraded
incumbent keeps its seat but can never be named host:
host_invalidflags the seated-but-incapable host (unit-pinned including the Direct-endpoint loss), the downgrade-reconnect re-election re-emits fresh plans end to end over real sockets (host_downgrade_reconnect_reelects_and_empties_downgraded_plan), a departure heals a wedged entry (non_host_departure_heals_present_but_unpairable_host), and an aborted re-plan retains the entry for the next event (aborted_replan_transaction_retains_the_wedged_entry_for_the_next_event). - Publication order. The
start_game_publication_buildermechanism queuesGameStartingas phase zero and each per-recipientSessionPlanas phase one in one exact-membership transaction; finalized joins and reconnects queue the actor's plan ahead of incumbents' in two ordered phases (actor_close_after_commit_does_not_suppress_incumbent_plan_phase); signal dispatch holds the room mutation gate so noSignalcan overtake a recipient's plan (signal_dispatch_waits_for_room_plan_publication_gate,signal_waiting_on_plan_gate_cannot_cross_target_incarnations).
With this review, the gameplay-transitions case families have recorded
dispositions. The coverage row moves to partially reviewed: the shared
src/server.rs state seams remain for later slices.
C1 room-event duplicate-delivery disposition, #713 (2026-10-01)¶
Issue #713
reported that, while iterating on the spectator-start pin, two awaited
handle_player_ready toggles delivered each LobbyStateChanged twice per
member when per-frame recv awaits were interleaved between the operations,
and asked whether the room-event lane's lease/stall recovery re-runs a
completed publication when tokio's auto-advanced paused time trips a lease
deadline. Investigated at 97506b11 (main after #714). Disposition: the
suspected mechanism does not exist and the doubling was not reproducible in
any arrangement; no defect is confirmed.
- The lane has no lease deadline and no re-run path. The room-event
mutation lease is a plain
tokio::sync::Mutexguard (RoomEventMutationLease,src/coordination/mod.rs) with no TTL and no recovery; the ready-toggle publication takes no distributed lock. The slow-consumer delivery deadline arms only under per-recipient backpressure and, on expiry, disconnects that recipient — it never re-runs the publication (reserve_one_if,src/server.rs).RoomEventLane::drainexecutes each enqueued job exactly once: the queue pop removes the job, the mutex-guardedrunningflag keeps at most one live drain per lane, and the completion resolves only after the job's sends finish. A completed publication cannot re-run. - The reported arrangement cannot advance the paused clock. After an
awaited toggle, every broadcast frame is already queued, so interleaved
recvexpects never park and tokio's auto-advance never fires. No timer-driven mechanism can intervene between the operations at all. - Non-reproducibility sweep. Five mechanically distinct arrangements
delivered exactly one
LobbyStateChangedper toggle per member: awaited toggles with interleaved expects (paused), the same with a spectator join and a router-dispatchedStartGamerefusal interleaved (paused), spawned non-awaited toggles with interleaved expects (paused), a fully idle 30 s auto-advance window between awaited toggles (paused), and real time withoutstart_paused. All clean. The original scratch test was not retained, and a single-run observation remains unexplained; if the pattern reappears, capture the exact test. - Pinned invariant.
interleaved_awaits_deliver_each_lobby_broadcast_exactly_once(src/server/ready_state_tests.rs) encodes the report's detection recipe: interleaved expects consume one copy per toggle, a fully idle window lets the paused clock auto-advance past any suspected deadline, and a final drain must find no second copy. Red-proofed by enqueueing the toggle publication twice in a probe: the pin fails on the duplicated frame. The probe was reverted byte-identically. - Test-authoring hazard recorded. During the sweep, a parked idle window
after a silently failed setup (a 5-character room code refused by the
fixed
room_code_lengthvalidation) surfaces a confusingError(NOT_IN_ROOM)at the next expect. Tests that drain instead of expecting can hide such setup refusals; expect the first frame of a phase when its arrival is the phase's evidence.
C1 cross-room stall fairness review (2026-10-01)¶
At 72d9188f (main after #715), reviewed the delivery slice's
"slow-recipient isolation; healthy-room progress during another room's
stall" case families. The within-room half was already pinned
(slow_consumer_no_cascade_e2e.rs, relay_backpressure_e2e.rs: one stalled
recipient is evicted loudly while its own room keeps flowing); the cross-room
half — one room's stall must not strand an unrelated room — had no
executable pin. The coupling audit found no shared state a stalled recipient
can hold across its delivery wait, and the invariant is now pinned.
Coupling audit (each shared seam a stall could hold, with dispositions):
- Room event lanes.
RoomEventLane::drainruns one job at a time per room; a lane job parked on a slow recipient delays only that room's later mutations. Other rooms get their own lanes; the sequencer registry stores weak handles only. - Backpressured data deliveries.
BackpressuredDeliveryowns per-recipient state only. The routed fan-out paths (start_routed_deliveries,start_routed_deliveries_with_shared) are synchronous under the routing snapshot and drop the sharedroom_playersandlocal_clientsread guards before any capacity wait is awaited; the parked wait holds no shared lock across its slow-consumer deadline. - Concurrent waits and eviction.
finish_deliveriesawaits all backpressured recipients concurrently (join_all), bounding one broadcast's latency to its slowest single recipient; the subsequent slow-consumer removal takes the evicted player's routing write gate and their rooms' gates briefly, with no await inside the critical section beyond lock acquisition. - Control-plane conditional deliveries.
reserve_one_if's capacity wait likewise owns only per-recipient state; the caller drops the shared guards before awaiting the reservation. - No shared egress budget. The delivery path contains no semaphore or
per-app budget a stalled recipient could exhaust: the only semaphore-class
primitives in
src/are#[cfg(test)]scaffolding, and the distributed room-cap lock (room creation) is mutex-based, not a delivery-path permit. - Sender tasks. A sender's connection task awaits its own room's fan-out (within-room ordering by design, bounded by the slow-consumer deadline); other rooms' connection tasks are independent tokio tasks.
Pinned invariant.
stalled_room_does_not_strand_a_healthy_room
(tests/cross_room_stall_fairness_e2e.rs) runs two independent rooms on one
server: room A holds a flooding sender and a never-reading recipient
(connected over a clamped 4 KiB receive buffer, so the wedge is deterministic
on every host), so its fan-out parks in backpressure until the slow-consumer
deadline evicts exactly that recipient; room B runs a continuous relay flood,
a member that joins while the stall-room flood is in flight, and two draining
recipients. When room A's sender observes the stalled peer's PlayerLeft,
room B must already have relayed frames through the whole grace window, the
join's PlayerJoined broadcast must have reached a room-B recipient, and no
fair-room member may ever observe a PlayerLeft, an Error, or a socket
close; the fair-room relay's longest inter-frame gap must stay under half the
stall window (a bare frame count is not enough — early frames before the
wedge would satisfy it); and exactly one slow-consumer eviction with
abandoned-frame drops must be counted. Red-proofed by holding a shared gate
across the game-data dispatch in a probe: room B's relay went silent for the
full stall window and the pin failed on the gap oracle; the probe was
reverted byte-identically.
Pin-authoring notes. Wedging a recipient deterministically requires (a)
frames under the 64 KiB inbound max_message_size — larger frames are
refused before fan-out and wedge nothing; (b) the heartbeat Pong deadline
above the delivery deadline — otherwise the heartbeat reaper evicts the
stalled peer first and websocket_slow_consumer_disconnects stays zero; and
© a clamped receive buffer on the stalled peer's socket (the repo-standard
connect_with_small_recv_buffer helper) — unclamped loopback autotuning can
absorb tens of MiB, the writer never parks, and no Full is ever reported.
With this review, the delivery slice's cross-room fairness family has a recorded disposition. The coverage row for relay routing moves to partially reviewed: mixed conversion refusal remains.
C1 reconnect epoch and sequence review (2026-10-01)¶
At a096bf33 (main after #716), reviewed the delivery slice's "reconnect
epoch and sequence changes" case families. Disposition per family; no
violation was reproduced.
- Resumed epoch without a provisional value. The reconnect path folds
last_epoch + 1(the pre-disconnect epoch survives in the reconnection record) into the reassignment itself, so the first metadata read already observes the final incarnation and no transient-socket epoch is ever visible. Pinned (reassign_connection_applies_the_resumed_epoch_immediately,src/server/connection_manager.rs; e2ereconnecting_sender_bumps_epoch_and_stamps_it_on_game_data). - Recipient-visible stream ordering. A recipient that never left sees the
sender's
(epoch, seq)stream strictly increase across reconnect (epoch bump, seq restart at 1), rejoin (self-describing epoch jump), and room switch (epoch carries forward, seq restarts). Pinned (sender_leave_and_rejoin_restarts_seq_at_one,epoch_carries_across_a_room_switch_not_reset_to_one); the recipient's epoch attribution ridesPlayerReconnected.epoch(goldengolden_player_reconnected_with_epoch) and the room snapshots (e2ev3_room_snapshots_carry_epoch_pre_v3_omit_it). - Stale-sender dispatch racing the rekey. The reconnect rekey waits on
the sender's lifecycle gate, which is held across the budget charge, the
stamp, and the enqueue (issue-#686 fence,
src/server/game_data.rs); a stale socket's frame is dismissed with no charge. Dispositioned by the completed lifecycle-capture-point sweep (ARM-C030); the gate is released before backpressured fan-out completion, so the rekey never waits on a queue drain. - Epoch saturation. Both bump paths (
prepare_client_to_room, the reconnect resume) usesaturating_add, can never regress an epoch, and log loudly atu32::MAX(src/server/connection_manager.rs,src/server/reconnection_service.rs). The terminal-incarnation reuse is unreachable in practice (~2^32 incarnations of one sender) and is reviewed by inspection; no pin. - Failed-restore rollback.
restore_reassigned_connectionresets the roomless transient identity's stamp state to 0, dominated by the reconnect-path epoch resume on the retry. Dispositioned by the failed-restore and retry review above (2026-09-30). - Cross-epoch exact gap accounting. Each incarnation restarts
seqat 1, so a sender's loss ranges from different epochs overlap numerically; a merge that ignoredepochwould swallow the newer incarnation's loss record into the older epoch's range and the client would never learn of the newer loss. The single merge rule (gaps_merge,src/coordination/outbound_queue.rs) requires epoch equality and is shared by the queue's gap reports and the writer's pending unsupported-format report, so one clause guards both. Now pinned (cross_epoch_gaps_of_one_sender_stay_distinct_ranges): epoch-4 and epoch-5 supersession losses of one sender must surface as two distinct ranges, each carrying its own epoch. Red-proofed by removing the epoch-equality clause in a probe: the ranges collapsed into one epoch-4 range and the pin failed; the probe was reverted byte-identically. - Old-epoch frames queued across the reconnect. Fan-out appends in stamp
order and the data lanes drain FIFO, so a recipient's undelivered
old-epoch frames precede every new-epoch frame and no spurious gap is
reported for them; per-epoch loss attribution is the pinned merge rule
above, and cross-epoch supersession attribution is pinned
(
latest_key_spans_sender_epochs_and_reports_the_replaced_epoch).
With this review, the delivery slice's reconnect epoch/sequence family has a recorded disposition. The coverage rows for reconnection and coordination gain the new evidence; their remaining items are unchanged.
C1 latest coalescing keys and generations review (2026-10-01)¶
At b851bf50 (main after #717), reviewed the delivery slice's "latest
coalescing keys and generations" case families. Disposition per family; no
violation was reproduced. One explicit-pin gap was closed.
- Key composition
(from_player, room_id, key). An equal application key from a different sender, routed through a different room, or carrying a different key value is an independent stream. Owner and room isolation are pinned (latest_supersede_requires_matching_stream_owner_and_room); the key value now has its own pin (latest_supersede_requires_the_matching_key_value): two keys of one sender in one room both deliver in enqueue order with no supersession counter movement and no gap report. Red-proofed by probinglatest_key()to a constant key: the second value superseded the first (losses: 1) and the pin failed on the enqueue outcome; the probe was reverted byte-identically. Cross-epoch, the key deliberately excludesepochso one logical stream continues across a sender's reconnect; the supersession report carries the replaced (older) incarnation's epoch (latest_key_spans_sender_epochs_and_reports_the_replaced_epoch). - Class/key contract at dispatch. A v3
latestframe withoutkey, areliable/volatileframe with one, and any class/key on a pre-v3 sender are refusedInvalidDeliveryClassbefore the payload cap and the relay-budget charge, before fan-out, and without consuming a relay sequence (pinned e2einvalid_delivery_class_does_not_consume_a_relay_sequenceover all four illegal pairings;pre_v3_sender_with_delivery_metadata_rejects_invalid_delivery_class; v2 wire tests). The queue'sInvalidMetadatafail-closed guard is defense in depth behind the dispatch gate (queue_rejects_control_data_and_mismatched_delivery_metadata). - Generation shielding. After a room transition, the coalescing scan
and the volatile-eviction scan match only current-generation rows: a
stale-generation same-key row is neither superseded nor evicted, stale
rows drain in fence order before the transition barrier, and a
dropped-full report lands after the barrier
(
room_transition_shields_stale_generation_rows_from_latest_scans). A stale-scope arrival is canceled byscope_matchesbefore any scan. - Supersession mechanics. Each supersession removes the exact
predecessor, emits its causal
LatestSupersededgap carrying the predecessor's(from_player, epoch, seq), and enqueues the successor with the key's original pendency time, so a continuously superseded key neither extends its coalesce deadline nor loses its causal report (latest_supersession_appends_successor_and_reports_exact_predecessor; e2elatest_coalescing_reports_exact_gap_before_successor; the anti-starvation bound is pinned bycontinuously_superseded_latest_key_still_reaches_the_socket). - Saturation paths. A latest arrival at a full data lane evicts the
oldest current-generation volatile with a causal
VolatileDroppedreport, never a reliable row; with no eligible victim the arrival drops withLatestDroppedFull(full_latest_evicts_oldest_volatile_before_dropping_arrival,volatile_replaces_oldest_volatile_but_never_reliable,sustained_latest_overload_coalesces_reports_without_fail_close); an unreportable loss fails closed (lossy_change_fails_closed_when_report_lane_is_full). - Coalescing window. Only a
Latestfront arms the batch window and it releases on window elapse, batch threshold, or queue progress; an interleaved control pop never consumes the armed budget; the pre-v3 legacy lane never coalesces and fails closed on aSome(Latest)row (regression_198_latest_behind_reliable_still_coalesces,interleaved_control_pop_preserves_armed_latest_batch_budget,pre_v3_data_never_arms_latest_coalescing_deadline,latest_row_on_legacy_lane_fails_closed_as_accountability_breach). - Counter conservation. Per-class counters conserve every terminal
outcome, including latest supersession
(
per_class_metrics_conserve_every_terminal_outcome).
With this review, the delivery slice's latest coalescing keys/generations
family has a recorded disposition. The coordination and queues coverage row
gains the new evidence and moves to partially reviewed: its delivery-side
queue families are dispositioned, while the reservation/commit
cancellation and panic seam of RoomMessageTransaction still needs its own
review.
C1 mixed encoding and unsupported conversion review (2026-10-02)¶
At 3b83e00e (main after #720), reviewed the delivery slice's "mixed
encodings and unsupported conversions" case families. Disposition per
family; no violation was reproduced. One explicit-pin gap was closed: the
pre-v3 recipient side of an opaque refusal had no end-to-end evidence.
- Negotiation boundary. A requested
game_data_formatoutside the deployment's supported set is refused with a budget-chargedUnsupportedGameDataFormaterror and downgrades to JSON, the universal text floor (test_rkyv_game_data_format_request_falls_back_to_json_and_is_not_advertised,test_opt_in_rkyv_and_protobuf_negotiate_and_advertise). Format negotiation is protocol-version-agnostic. - Same-format and lossless cohorts. A JSON sender reaches every
recipient as text; a MessagePack payload reaches same-format recipients
as a direct binary cohort and JSON recipients through a lossless
MessagePack-to-JSON decode, with no error amplification
(
mixed_json_and_message_pack_relay_without_error_amplification). - Opaque refusal. rkyv and protobuf payloads carry no schema the
server can convert, so every cross-format recipient is skipped instead
of receiving a lossy guess (
decode_binary_to_jsonrefuses both). A same-format peer still receives byte-identical strict-v3 envelopes; cross-format v3 peers receive an exactUnsupportedFormatgap report before any later frame plus the per-sender rate-limited advisory (opaque_opt_in_encodings_relay_directly_and_report_cross_format). The refusal is pair-complete: every directed pair from an opaque source is unsupported, and the preflight unit matrix pins the same decision per pair (unsupported_binary_fallback_preflight_is_exact). - Pre-v3 recipients (new pin). A v2 queue never accumulates a pending
unsupported report (
record_unsupported_formatrefuses non-v3 queues) and the report writer keeps its own v3 gate, so aDeliveryReportcan never leak onto a v2 wire. The omission is still counted (websocket_messages_dropped, per-connectiondropped_for_you) and the recipient still receives the rate-limited advisory, now pinned byv2_recipients_of_opaque_payloads_get_advisories_without_v3_reports: a v2 JSON and a v2 MessagePack recipient of one opaque rkyv frame each observe no payload frame, one advisory, no report, and a control plane that keeps flowing through the sender's leave. Red-proofed by probing the opaque refusal into a lossy JSON fabrication: both v2 recipients received the fabricatedGameDataframe and the pin failed; the probe was reverted byte-identically. - Oversized fallback. A cross-format payload whose decoded JSON would
exceed the outbound cap is refused by the preflight before cache
allocation and accounted as an omission
(
binary_fallback_decode_budget_rejects_compact_tree_before_cache_allocation). - Amplification under load. The throttled unsupported-format storm
scenario (a weaker recipient must not be evicted; advisories only) is
pinned by the nightly-only
unsupported_message_pack_fallback_does_not_flap_weaker_recipientand the report/advisory ordering conformance tests (conformance_gap_counters_are_causal_with_rate_limited_unsupported_advisories,conformance_unsupported_advisory_requires_prior_report_but_not_adjacency); the storm cohort stays out of the default CI run by design.
With this review, the delivery slice's mixed encodings and unsupported conversions family has a recorded disposition. The relay-routing coverage row is fully reviewed.
C1 permitted volatile loss and exact gap/report accounting review (2026-10-02)¶
At b4d30b1a (main after #723), reviewed the delivery slice's "permitted
volatile loss" and "exact gap/report accounting" case families. Disposition
per family; no violation was reproduced. Two composed-path pin gaps were
closed with one real-socket pin.
- Queue-level lossy drops. Supersession, full-lane volatile eviction,
and dropped-full arrivals each drop at most one row per lossy enqueue and
emit a causal exact gap (
try_enqueue_latest,try_enqueue_volatile): the matrix is pinned at queue level (latest_supersession_appends_successor_and_reports_exact_predecessor,full_latest_evicts_oldest_volatile_before_dropping_arrival,volatile_replaces_oldest_volatile_but_never_reliable,room_transition_shields_stale_generation_rows_from_latest_scans,sustained_latest_overload_coalesces_reports_without_fail_close), and the supersession half is pinned end to end (latest_coalescing_reports_exact_gap_before_successor). An unreportable loss never drops silently:fail_accountabilityfails the whole queue closed (lossy_change_fails_closed_when_report_lane_is_full), so loss can never outrun its report capacity. - Coordinator fan-out drops. Accounted drops land in both the server-wide
counter and the per-connection ledger (
record_queue_outcome); accountability-unavailable closes the connection loudly; a cancelled park resolves attempted+abandoned without closing (the recipient is not at fault, and latest/volatile loss is part of the delivery model). - Teardown and writer abandonment. Abandonment counts per class from the
queue and batcher, and the abandoned-in-flight fence abandons the remainder
rather than write a hole no report covers
(
close_flush_never_writes_the_queue_behind_an_abandoned_write): the observable stream stays a gap-free prefix plus exact reports. - Generation fences. A stale-scope lossy arrival is canceled
(CANCELED,
websocket_deliveries_canceled), not dropped-and-reported: the epoch bump plus transition barrier make the boundary self-describing, and stale rows drain before the barrier. - Report exactness mechanics. Lossy enqueues reserve causal report
capacity before mutating the data lane (
can_record_gap); the single merge rule requires epoch equality (cross_epoch_gaps_of_one_sender_stay_distinct_ranges); frontier stamps clamp to what was written and commit only after flush; exact ranges are never throttled while the advisory prose is limited to 1/s per sender; the 256-range bound rolls over; the v3-only gates hold at record and write time. Each is pinned (queue and writer unit tests, the conformance auditor's counter-delta/range-validity oracles, and the golden wire tests). - Persistence across reconnect. Reports do not persist across reconnect
by design: a resumed recipient is baselined by authoritative
SenderWatermarks, and the cumulative per-connection ledger survives a rekey. Permitted silent drops are limited to advisoryRelayStatsframes on a full control queue (cumulative counters; the teardown is the signal) and the farewell advisory skip on a connection that is closing anyway. - New pin (closes both composed-path gaps). No PR-lane test had observed
a real volatile eviction over a socket, and no test had observed the
per-connection
dropped_for_youledger move off zero.flooded_nonreading_recipient_observes_exact_volatile_gaps_and_dropped_for_you(tests/v3_game_data_sequencing_e2e.rs) floods 2,000×16 KiB volatile frames at a silent recipient whose data lane is two slots: the sender is never backpressured and the recipient is never closed; the observed seqs plus the exact disjointVolatileDroppedranges must cover the whole offered stream; the cumulativevolatile.droppedcounters (wire report and class ledger) must equal the missing count;dropped_for_youmust be non-zero in both theRelayStatsframe and the server-side connection ledger; and a marker frame must arrive with the stream's next seq. Red-proofed by suppressing the causal gap report at the volatile-eviction site: coverage never closed (delivered=19, gaps=[]with 1,981 counted drops) and the pin failed; the probe was reverted byte-identically. The class-ledger conservation read settles before asserting equality: the writer records a delivered row after its socket write resolves, so the ledger can lag the recipient's observation by a scheduling beat (first CI run caught exactly that sampling race, 18 counted vs 19 observed).
With this review, the delivery slice's permitted-volatile-loss and
gap/report-accounting families have recorded dispositions. Reviewed by
inspection with no separate pin: the advisory limiter's oldest-sender
eviction at its 256-sender cap (the bound itself is pinned by
unsupported_notice_limiter_bounds_sender_state), and the metadata-less
record_unsupported_class branch that sits behind the fail-closed arm.
C1 transaction reservation/commit cancellation and panic review (2026-10-02)¶
At 9e07ecd7 (main after #724), reviewed the recovery slice's opening seam:
cancellation and panic at RoomMessageTransaction reservation and commit
(commit_room_messages_if_members_with_hook and its sibling
broadcast_to_room_if_with_hook). One defect class was reproduced and fixed;
every case family now carries a disposition.
- Structured cancellation at the transaction's awaits. Every pre-hook terminal path (batch canceled, slow consumer, channel closed, routing change, hook rejection, hook error) already recorded each reserved frame and retried or returned explicitly; the lane's spawned jobs cannot be dropped by a caller (detached-job pin at the lane level), so the only caller-visible cancellation is process teardown (drain row).
- Red-first defect: panic released reservations without accounting. A
panic inside the commit hook, the phase callback, or the broadcast replay
hook unwound through its publication path and dropped every held permit
silently: no
websocket_deliveries_canceledmovement, while the same terminal paths reached byErr/falseeach count exactly once. Red evidence:panicking_commit_hook_releases_and_accounts_every_reservationfailed withleft: 0, right: 4andpanicking_phase_callback_accounts_remaining_frames_and_never_delivers_phase_onefailed withleft: 0, right: 1on the unfixed code; the broadcast replay-hook pin was red-probed against the fixed code by disablingConditionalReservationGuard's accounting flag (left: 0, right: 2); the probe was reverted byte-identically. - Fix (class sweep). Reservation ownership on both publication paths now
flows through drop-accounting guards (
RoomBatchReservationGuardand a siblingConditionalReservationGuardof the same design) that count each still-undelivered reserved frame exactly once on any exit that is not an explicit, accounted terminal path. The two explicit helper functions and all twelve manual call sites were removed; the guards cover the window from guard arm (immediately after the batch reservation resolves) through commit, so a missed accounting exit is unreachable there, including for future early returns. The commit loop consumes permits through the guard, so its own per-frame accounting never double-counts. - New pins (one per fixed seam). The three pins (two red-first, one
red-probed) cover each fixed seam.
panicking_commit_hook_releases_and_accounts_every_reservation: a panicking commit hook delivers no frame, counts all four reserved frames exactly once, releases recipient capacity (a follow-up transaction's worth oftry_sendsucceeds), and surfaces as the job's panic.panicking_phase_callback_accounts_remaining_frames_and_never_delivers_phase_one: a panicking phase callback delivers phase zero exactly once, never delivers phase one, counts the unpublished phase-one frame, and releases its capacity.panicking_broadcast_replay_hook_releases_and_accounts_every_reservation: a panicking broadcast replay hook delivers no frame, counts both reserved frames, and releases their capacity. All three bind the observed panic to its sentinel message, not just tois_panic. - Panic recovery shape (by inspection, no new pin). Each panic surfaces
as the room-event job's
JoinError, which the lane isolates (pinned,panicking_room_event_isolated_from_the_next_job) and converts to a caller error, so each caller's existing publication-failure arm (for example the join fallback) runs unchanged. A panic after the hook's durable mutation commits leaves phase zero or nothing published; the reconnect baseline is the client-driven recovery path, matching the recorded degraded-restore disposition. The broadcast pin drivesbroadcast_to_room_if_members_with_hook, whose wrapper differs from the production replay-hook callers only in membership filtering before the same guarded core. - Sibling sweep.
broadcast_to_room_if_with_hookhad the same silent window inside its replay hook; its reservations now flow through a sibling guard of the same design, and its drain/continue/rejection paths keep their exact prior accounting semantics.reserve_one_if's parked waits were already fenced (ParkedWaitAccounting, #417).
With this review, the coordination and queues coverage row is fully reviewed: its remaining reservation/commit cancellation and panic families carry pinned evidence and the row moves to reviewed. The recovery slice continues with deadlines, wall-clock versus monotonic expiry, drain/shutdown with queued data, and process-loss behavior (admin/shutdown coverage row).
C1 maintenance and deadlines expiry boundary review (2026-10-02)¶
At 98d86306 (main after #725), reviewed the recovery slice's
deadlines-at-boundary and wall-clock-versus-monotonic families across
src/server/maintenance.rs, heartbeat.rs, dashboard_cache.rs, and
src/deadline.rs. No defect was found; the one unpinned player-visible
boundary now carries a pin, and every family carries a disposition.
- The activity-reaper pair flips once at the boundary (new pin).
collect_expired_clientsandrequest_activity_timeout_if_expiredboth read monotonictokio::time::Instantand agree at exactlyping_timeout: the snapshot does not collect the client and the atomic revalidation refuses to pin the close, while one tick later both expire and the revalidation pins theActivityTimeoutclose. The pairing is what lets a Pong arriving between the snapshot and the revalidation rescue the connection (the entry-guard exclusion documented on the revalidation). New pinactivity_reaper_expiry_flips_once_at_the_ping_timeout_boundary(paused clock), red-proofed by flipping the snapshot comparison to>=and the revalidation to<: the pin failed at the exactly-at survival assertion; the probe was reverted byte-identically. - Zero timeout disables the reaper end to end (pinned, existing). The
cleanup task's
ping_timeout.is_zero()guard short-circuits the snapshot, pinning the documented "0disables the activity reaper" contract (zero_ping_timeout_disables_activity_reaper). The revalidation predicate has no internal zero guard by design; its only production callers sit inside that guarded loop. - Monotonic decision paths (pinned, existing). Reaper, throttle,
reconnect window, room GC, dashboard staleness, and the cleanup-claim
window all decide on the runtime clock, and
tests/clock_source_scan.rswalls production time reads behind injectable or tokio seams. Boundary pins:should_update_last_seen_throttles_until_threshold_elapses(inclusive by design for the last-seen database-write throttle),reconnect_eligibility_flips_once_at_the_monotonic_deadline(strict),wall_clock_step_cannot_reap_monotonic_fresh_occupied_room,monotonic_idle_rooms_are_reaped_despite_fresh_wall_stamps,every_activity_path_refreshes_monotonic_liveness,cleanup_claim_and_prune_boundaries_run_on_monotonic_time, andstaleness_gate_decides_on_monotonic_elapsed_time(strict). - Wall-clock changes cannot open or close a monotonic deadline.
Reconnect records capture both clocks at the same moment; a wall-clock
jump rewrites every stored UTC field but not the decision deadline
(
wall_clock_jumps_cannot_open_or_close_the_reconnect_window). Room rows pair the walllast_activitystamp with the monotonic liveness stamp, and the wall fallback exists only so a foreign row insertion cannot become an immortal room. The drain's advertised wall-clock deadline is clamped by one grace period for later observers (wait_before_close_since), so an overflowed sentinel or a backwards step cannot stretch the drain (in-file pins). The publicReconnectionToken::is_expired/is_validwall-clock answers are embedder conveniences documented as not the admission decision; verified to have no server-internal caller. - Deadline overflow never inverts into immediate expiry (pinned,
existing).
deadline::after/saturating_after/wait_untilare pinned in file and consumed by the backpressure, reconnect-window, ping-write, and shutdown-wait seams; the shutdown deadline arithmetic saturates, and the dashboard's staleness conversion saturates (chrono::Duration::from_std(..).unwrap_or(chrono::Duration::MAX)). - Sibling sweep of expiry predicates (recorded dispositions).
Rate-limit window (inclusive,
window_admits_up_to_limit_then_expires_at_the_inclusive_boundary), batching write deadline (inclusive,selected_write_expires_at_or_after_deadline_without_completing_accounting), room-state expiry at caller-injected now (is_expired_at_times_windows_off_last_activity_at_the_callers_now), and the database room-idle comparisons (strict, consistent with the in-memory reaper) are each pinned or inspected-consistent. TheInMemoryDistributedLocklease predicates uniformly readexpires_at > nowwith no dedicated fail-open/fail-closed branch; covered off-boundary by real-time tests and recorded here by inspection, since an exactly-at early expiry errs toward releasing a lease, the safe direction for a coordination lock and not player-visible.
The maintenance and deadlines coverage row moves to partially reviewed: the expiry-boundary and clock-source families are reviewed and pinned, while churn growth and dashboard cost remain measurement work for C3/C5. The recovery slice continues with drain/shutdown racing active reconnect claims and queued data, cleanup racing join/reconnect, and process-loss behavior.
C1 drain/shutdown with queued data and active reconnect claims review (2026-10-03)¶
At 54feb610 (main after #726) with this review's fix, reviewed the
recovery slice's drain/shutdown family across src/server/shutdown.rs,
connection_manager.rs, reconnection_service.rs, room_service.rs,
spectator_service.rs, the coordinator's initial-transition registration
(src/server.rs), and the per-connection close path
(src/websocket/connection.rs). Two accounting defects were found and
fixed; every family carries a disposition.
- A committed reconnect claim always reaches the close fan-out
(fence verified).
begin_shutdown_draintakesshutdown_drain_commit_gatebefore flipping the drain atomic, so a baseline commit either finishes under the gate before the flip or seesshould_commit() == falseand refuses. The restored identity is inserted intoconnection_managerbefore the coordinator commit, so the laterclose_connections_for_shutdowniteration sees it; a fan-out that ran before the swap pins the transient socket, and theShutdownreason is classified as crossing the identity swap and is carried into the restored connection's close signal (pinned,reassign_still_adopts_entry_pinned_for_shutdown). New upgrades are refused while draining, with a post-registration recheck pinning the inline 4000 if the flip races the insert. The committed-claim end state —Reconnectedbaseline, then the coded 4000 — is therefore guaranteed and is pinned piecewise: the wire shape byshutdown_drain_sends_goingaway_and_closes_4000_without_reconnect_recordand the swap-crossing pin above. - A drain flip during a parked baseline reservation was misaccounted
(fixed, new pin). The initial-transition reservation's backpressure
park had no drain arm, so the outer drain race dropped the parked future
and the parked-wait guard resolved the attempt as
websocket_messages_dropped— the sibling conditional-delivery park records the identical event aswebsocket_deliveries_canceledwith an explicit not-a-drop pin. The reservation park now carries the same biased drain arm (cancellation precedence over capacity and expiry), and the pre-attempt refusal moved into the reservation itself (pinned bypre_flipped_drain_refuses_the_initial_transition_before_counting). New pindrain_flip_during_initial_transition_park_cancels_instead_of_dropping, red-proofed by reverting the fix: the flip counted a dropped message and left the cancellation ledger untouched. - Two silent releases of a reserved baseline (fixed, new pins). A
failed baseline build and a drain-gated commit refusal each released the
reserved initial frame with no paired outcome for the counted attempt —
the same class the #725 panic-accounting fix closed for transaction
hooks. Reservation ownership now flows through
InitialTransitionReservationGuard, defused only when the commit resolves the attempt.drain_gated_commit_refusal_accounts_the_reserved_baselineandfailed_baseline_builder_accounts_the_reserved_transitionanddropped_registration_between_reservation_and_commit_accounts_the_attempt(external cancellation mid-build) are red-proven against the unfixed revision;committed_initial_transition_resolves_its_attempt_exactly_oncepins the no-double-count property on the committed path. Release-path pins assert the freed queue slot with a follow-up send on a one-slot channel. - Close ordering and queued data (contract-consistent, pinned
piecewise). The close frame is written by the per-connection send task
after a bounded flush (one second per close step), so a coded 4000 never
overtakes queued frames; a remainder past the budget is abandoned and
counted, never written late. A cancelled mid-write frame makes the
remainder abandoned instead of flushed (a gap-free prefix that stops
early; a hole never). Per the protocol contract, messages abandoned
during close need no gap records because the recipient's own disconnect
terminates the observable stream; the conservation identity
attempted = delivered + abandoned + unsupportedholds per class, and a pending unsupported-format report is flushed before the close frame. - Shutdown closes are terminal for reconnection state (pinned,
existing). A draining unregistration skips the reconnection record,
discards the pre-issued token, and hard-removes the room row; a record
registered just before the flip is discarded when the removal observes
the
Shutdownreason (draining_unregister_discards_reconnect_when_drain_starts_during_leave,shutdown_drain_sends_goingaway_and_closes_4000_without_reconnect_record). Replay rings are instance-local memory, so a restart cannot resume them; the restart-invalidates-tokens behavior is pinned in the multiprocess suite. - Shutdown-close observability (recorded disposition). The drain's
close fan-out is logged but there is no exported counter distinguishing
code 4000 closes; upgrade refusals during drain are counted
(
websocket_upgrades_rejected_draining) and abandoned queue frames are counted per class. A dedicated 4000-close counter is filed as follow-up observability work (#727), not a correctness defect.
The admin and shutdown coverage row moves to partially reviewed: the drain choreography, its commit fence, queued-data close ordering, and reconnect interactions are reviewed and pinned or fixed, while distinct-metric export remains follow-up work. The recovery slice continues with cleanup racing join/reconnect and process-loss behavior versus documented limits.
C1 authentication admission boundary review (2026-10-03)¶
The tolerated source-charge race below is superseded by F15 in the production correctness audit. The earlier experiment established ceiling conservation, not rejected-source retry availability.
At d70867db (main after #734), reviewed the resource-and-input-safety
slice's unauthenticated-admission families across src/auth/middleware.rs,
src/auth/rate_limiter.rs, the room-side budgets in src/rate_limit.rs,
and the connection receive loop's handshake handling
(src/websocket/connection.rs). No defect was found; every family carries
a disposition and the two named remaining cases are now pinned.
- Unauthenticated flood posture (verified). Every
Authenticatespends the app's handshake windows before credential verification (resolve_app_idprecedesverify_connect_token), and any resolution or token refusal breaks the receive loop and closes the socket — a refused peer cannot retry on the same connection. Pre-handshake application frames are refusedMISSING_APP_IDand close the connection; the log-safety gate rejects hostile app IDs before any policy path; app-ID length is capped (MAX_APP_ID_LENGTH). - Auth timeout boundary is absolute (pinned,
pre_handshake_activity_does_not_extend_the_auth_deadline; red-proofed by making the deadline slide per read). The pre-handshake deadline is frozen atconnection_start + auth_timeout_secs; protocol-level keep-alives and other inbound frames neither extend it nor reclassify the cut as idle. The cut keeps code 4001 /auth_timeout, and a cut with received frames correctly stays out of the zero-frame disconnect counter. Config bounds (5–60s, no disable) were already pinned; the deadline is the only admission-bound deadline in the receive loop (the ping-write and pong deadlines are per-operation caps by contract, and the idle window is intentionally activity-reset). - Concurrent admission conserves the ceiling (pinned,
concurrent_handshakes_conserve_the_app_ceiling_and_count_every_rejection; red-proofed by disabling app-window enforcement). 8 sources × 2 racing handshakes against a ceiling of 4 admit exactly 4 under the worst-case probe/commit race; each of the 12 rejections increments the aggregate and auth rejection counters exactly once. The enforcement rests on the commit-time re-check under the limiter entry lock (concurrent_rate_limit_enforcementpins the primitive); a rejection never charges the other window, and the documented worst case is one wasted stamp in the offending source's own window under a probe/commit race (the #502 split-budget pin covers the sequential composition; this pin covers the concurrent one). - Room-side budgets (verified, previously reviewed). The room
creation/join/signal/relay budgets check-and-charge atomically under
the entry lock (
try_*/chargeonRateLimitEntry), the error-reply gate charges under the connection's own serialization, and the budget/refusal accounting pins from the earlier budget and error-reply sessions hold.PlayerRejectionStatsforensics are saturated-add and reset with the window.
The Authentication coverage row moves to reviewed: unauthorized traffic cannot enter a room, limits count refusals exactly once, concurrent admission conserves the ceiling, and the auth timeout boundary is absolute and activity-immune. The resource-and-input-safety slice continues with queue/replay bounds, parser boundaries, and metrics label cardinality.
C1 metrics label cardinality review (2026-10-03)¶
At 85abff77 (main after #735), reviewed the resource-and-input-safety
slice's metrics-label-cardinality family across src/metrics.rs, the
Prometheus renderer (src/websocket/prometheus.rs), and the bounded
/metrics dashboard snapshot (src/websocket/metrics.rs). No defect was
found; every label surface carries a documented bound with a pin or tested
lifecycle wiring.
- Per-connection ledgers are registration-scoped (verified, pinned).
connection_delivery_statsandslow_consumer_eviction_attributionskey by live player id: registered at connection registration (src/server/connection_manager.rs), re-keyed in both directions across a reconnection identity swap, and removed at unregistration.attribution_ledgers_accumulate_rekey_and_unregisterpins the accumulate/rekey/unregister/no-resurrection lifecycle, including the zero-ledger-exists-only-for-bounding rule. - Per-app relay attribution is allowlist-bounded (verified, pinned).
app_relay_bytesrecords only senders with an app policy (check_and_charge_relay_bytescharges underif let Some(policy)), so open-mode client-chosen app IDs never create series and the map is bounded by the allowlist.allowlist_reload_prunes_revoked_app_relay_seriespins the #552 pruning of a revoked tenant's series on allowlist reload. - Client-chosen game-name maps are response-bounded (verified).
roomsByGameandgamePercentilesare derived from the dashboard view per request and truncated bybound_response_game_mapon both the current view and every history sample (issue #518); nothing accumulates them server-side. - Shutdown closes are counted at the coded close fan-out (new, #727).
websocket_shutdown_disconnectscounts registered connections torn down with the server-initiated 4000 close, classified at the semantic close step after every reason override, so the drain close fan-out is observable without log scraping. The drain pin proves one closed connection counts exactly once, and the activity-reaper pin proves a non-shutdown close cannot land in the counter (tests/close_code_semantics_e2e.rs). A late registration refused during a drain never registered, so it stays underwebsocket_upgrades_rejected_draining.
The metrics label cardinality family is closed with bounded-series dispositions. The resource-and-input-safety slice continues with queue/replay bounds and parser boundaries.
C1 parser boundaries and queue/replay bounds review (2026-10-03)¶
At 48bdc7fb (main before this change), reviewed the
resource-and-input-safety slice's parser-boundary families (depth, size,
malformed frames, Unicode, numeric boundaries) across the ingress parsers
(src/websocket/token_binding.rs, src/protocol/binary.rs, serde_json
typed envelope decodes) and the conversion path
(src/websocket/sending.rs). One hardening was landed with differential
evidence: ARM-C030.
- Depth (hardened). rmp-serde decode into a recursive target leaned on
that dependency's internal 1024-level guard, not a wire contract; the
boundary is now an explicit 128-level scan before any recursive decode
(ARM-C030 for the rationale, differential evidence, and pins). The same
class was swept: the v3 binary envelope decodes through flat
rmp::decodefield reads, the JSON envelope decodes under serde_json's 128-level limit, and the token-bound binary envelope shares the scanner. rkyv and protobuf payloads are opaque to the server by design and are never decoded. On production 2 MB stacks no abort was reachable through the locked rmp-serde 1.3.1; the hardening removes the dependency-version dependence. - Size. The transport caps inbound frames at
2 × max_message_sizeand the receive loop re-checksmax_message_sizebefore parsing; per-encoding payload ceilings (security.max_game_data_bytes, #634) gate game data at admission; the conversion preflight bounds the decode input against the outbound budget before any allocation (binary_fallback_decode_budget_rejects_compact_tree_before_cache_allocation). - Malformed frames and numeric boundaries. Every malformed JSON,
MessagePack, and binary envelope returns a classified parse error with a
budget-charged refusal; fatal classes disconnect with a farewell, other
classes keep the connection. MessagePack integers outside the value domain
and non-UTF-8 strings are decoder errors; the strict v3 envelope pins
exact field types (
src/protocol/binary.rstests). - Unicode. Player names canonicalize through
icu_casemapandunicode-normalizationwith length caps (identity slice); wire strings are UTF-8-validated by both decoders before use. - Queue and replay bounds. The delivery-side queue families (bounds, saturation, coalescing, counter conservation) carry their dispositions in the delivery reviews above; the reconnect replay ledger is bound by the reconnect epoch/sequence review (2026-10-01) and the drain review (2026-10-03). No new violation was reproduced this review.
With this review, the parser-boundary families have recorded dispositions. The resource-and-input-safety slice continues with inactive records, pending detach/claim retention, task ownership, rate-limit rejection accounting, and error and logging paths under pressure.
C1 inactive records, pending detach/claim retention, and task ownership review (2026-10-04)¶
At dd37153b (main after #737), reviewed the resource-and-input-safety
slice's inactive-record, pending detach/claim retention, and task-ownership
families across the connection manager, reconnection service and manager,
maintenance sweeps, spectator service, moderation, the WebSocket connection
loop, and the shutdown/drain paths. No defect was found; every audited
record and task carries a verified removal or abort on all exit paths.
- Inactive records (verified clean). Every audited map is bounded by a paired lifecycle: connection admission slots release exactly once on unregistration (with a loud saturated underflow guard); coordinator routing maps prune empty room sets and re-sync the active set on every mutation; client-supplied labels never key a server-side map (session plans, ready sets, room applications, and app relay series are server-keyed with maintenance prunes or allowlist-bounded; the upgrade-rejection log evicts deterministically). The reconnection replay/release families carry their dispositions in the delivery, epoch/sequence, and drain reviews above.
- The pre-issued-token teardown branches are covered by layered
discards (hypothesis falsified). The teardown chain in
unregister_client_lockedhandles draining, snapshot-registration, and no-room discards, and its snapshot-unavailable branches (Ok(None), roster miss, storage error) fall through with no explicit discard. A review hypothesis claimed each such exit leaks the pre-issued token past teardown. The hypothesis is false: the same unregister flow always reaches the room-removal path, which discards the pre-issued token after the player leaves the room (room_service.rs), and the maintenance sweep independently cleans up clients whose room is missing. The pin (unregister_snapshot_failure_creates_no_broken_reconnect_record) already asserts both the pending-record absence and the pre-issued-token discard under a snapshot storage fault. - Pending detach/claim retention (verified clean). Every
pending_durable_player_detachesinsert has a removal: direct resolution, reconnect reclaim, or the maintenance retry sweep (storage success, room deleted, or live reclaim). Spectatorpending_unpublished_detachesentries resolve through owned rollback, republished-identity checks, or the retried sweep, and re-queue on read failure. Claim records are single-owner withclaim_idverification on every mutation; every expiry surface filters out claimed records so an in-flight claim cannot be swept, and every rejection path funnels through the claim rollback. - Task ownership (verified clean; the one theoretical residual is now
fixed). Per-socket
send/receive tasks are joined; the ping and relay-stats tickers exit on
the socket close signal that every unregistration requests. Server-level
tasks are owned: the drain task is joined or aborted at shutdown, the
cleanup loop is aborted on cancellation via a task-abort guard, the
dashboard-cache and rate-limiter sweeps hold
Weakowners and self-terminate, and lease renewal aborts on drop. The theoretical owned reconnect-transaction residual (#738, ARM-C037 below) is fixed: the in-task unwind supervisor is itselfcatch_unwind-guarded with a direct claim-release fallback, pinned bysupervisor_panic_still_releases_the_reconnect_claim_for_a_fresh_retry.
With this review, the inactive-record, pending detach/claim retention, and task-ownership families have recorded dispositions.
ARM-C037 — A supervisor panic could strand the reserved reconnect claim¶
| Field | Record |
|---|---|
| State, severity | Fixed, low (theoretical: required a second-order unwind that production code could not produce; filed as #738 from the 2026-10-04 task-ownership review) |
| Player impact | If reached, permanent: a claimed reconnect record is exempt from every expiry surface (expired_cleanup_candidates, remove_expired_reconnection, cleanup_expired all filter record.claim.is_none()), so the disconnected player's every reconnect attempt would answer AlreadyInProgress until process restart — a stuck seat plus a consumed one-time token. |
| Source and revision | src/server/reconnection_service.rs::spawn_reconnect_transaction, reviewed at dd37153b. The transaction's catch_unwind covered handle_reconnect_owned, but the in-task unwind supervisor arm that handles its panic ran OUTSIDE that guard while the recovery snapshot (ReconnectPanicRecovery) may already hold the reserved claim — the sole claim reservation sits inside handle_reconnect_owned. An adversarial falsification pass confirmed the pre-supervisor region is panic-free and nothing aborts the inline-awaited JoinHandle, so no production path reached the strand today; the invariant was hygiene-by-convention across the supervisor's callee tree, not a structural property. The join seam's stranded creation lock self-heals via the distributed TTL lease (src/distributed.rs); the leave seam reserves nothing — the reconnect claim was the only permanent-strand surface. |
| Invariant | A panic anywhere in an owned transaction's recovery path must not leave reserved, expiry-exempt state stranded; the recovery layer is part of the panic surface it manages. |
| Confidence and reproduction | Red-first: a one-shot test seam (panic_owned_room_supervisor_for_test) fires at the top of the supervisor body while ReconnectAfterReassignment has already populated the snapshot's claim. Pre-fix, the supervisor panic escaped as a JoinError, the log-only arm returned, and the fresh-retry claim_reconnection failed (supervisor_panic_still_releases_the_reconnect_claim_for_a_fresh_retry red at the retry expect). An earlier outer-JoinError-arm recovery shape was rejected: holding the snapshot Arc across task.await extends the snapshot's RoomEventMutationGuard hold past the task boundary and deterministically broke correlated_reconnect_panic_after_terminal_does_not_send_a_second_result — recovery must stay inside the task. |
| Disposition | The supervisor body is now wrapped in its own AssertUnwindSafe(..).catch_unwind(). On a supervisor panic it logs Owned reconnect transaction unwind supervisor panicked and resolves the record by the transaction's commit state, mirroring the supervisor's own branch: a committed terminal response consumed the one-time token, so the record is completed (consumed); an uncommitted transaction leaves the token unspent, so the record is released for a fresh retry. Both manager actions are claim-id-checked with warn no-ops when the supervisor already acted before panicking. The full rollback is deliberately not retried — code that just panicked can panic again — and the record actions are panic-free (tokio-lock based, poison-tolerant snapshot read). The outer task.await error arm stays log-only: with the supervisor guarded, reaching it requires runtime-shutdown cancellation, where all in-memory claim state dies with the process. PR #743 review (Cursor Bugbot) caught the fallback's first shape releasing unconditionally, which would have reopened a delivered token; red-proven by supervisor_panic_after_terminal_consumes_the_delivered_token beside the fresh-retry pin. Pre-existing residual noted, not introduced here: a panic between complete_claimed_reconnection and the opening_accounted swap skips the players_joined increment; left to the metrics row. |
C1 rate-limit rejection accounting review (2026-10-04)¶
At dd37153b (main after #737), reviewed the resource-and-input-safety
slice's rate-limit rejection accounting family across
src/rate_limit.rs, the auth sliding windows, the error-reply gate, the
join/spectator/signal lanes, relay byte charging, and connection admission.
No defect was found: every refusal path charges exactly once, every counter
lands on the budget that refused, and no path double-charges or inflates.
- Charges and counters are paired exactly once (verified). The compound room-creation charge is both-or-neither; the relay charge is all-or-nothing per budget with the room-ceiling rejection deliberately retaining the sender charge (pinned); the signal preflight never consumes and records a rejection only when it owns the drop; every retryable handshake, parse, moderation, authority, and drain refusal charges the error-reply gate exactly once via the charged helpers, and farewell/terminal refusals never charge. Auth windows keep rejections free via probe-before-commit, with the documented one-self-tightening race still counting every rejection (pinned). Admission refusals consume no slot, so no budget is consumed unaccounted.
- The drain-window creation refusal is deliberately budget-free (now
pinned). The create-while-draining fast-path runs before the
join/creation budget charge, while everything else refused in the same
window spends its bucket. The drain comment shows the choice is
deliberate: the drain-window flood bound is the charged error-reply gate
(issue #518), not the join bucket. The asymmetry is now pinned by
extending
draining_server_rejects_room_creation_without_consuming_join_locks: the drain-window creation refusal creates no budget entry, and an existing-room join admitted in the same window spends exactly one join attempt.
With this review, the rate-limit rejection accounting family has a recorded disposition.
C1 error and logging paths under pressure review (2026-10-04)¶
At dd37153b (main after #737), reviewed the resource-and-input-safety
slice's error/logging-under-pressure family across the WebSocket hot loop,
the relay send path, heartbeat, the messaging helpers, and the subscriber
in src/logging.rs. Two hardenings landed with this review (ARM-C031,
ARM-C032); everything else is verified bounded or charged.
- Rejected app-ID log forged log lines (fixed, ARM-C031). The
Public app ID rejectedwarning logged the raw client-supplied app ID with a Display field in exactly the arm where the log-safety gate had rejected it, so control characters in a rejected ID could forge operator log lines. The ID is now Debug-escaped, matching the pre-gate escaping pattern already used at the sibling anomalous-path warnings. A sweep of every remaining%app_idlog site found only gate-vetted or typed-UUID fields. - Undeliverable-relay warning flooded logs (fixed, ARM-C032). The
per-recipient warning in
notify_on_undeliverablefired once per undelivered frame per recipient with no throttle, while the in-band advisory it accompanies was already rate-limited to one per sender per second; one sender with an unsupported encoding into a large room could flood operator logs. The warning now rides the advisory cadence and carries the suppressed count. Delivery reports, advisories, and drop counters are unchanged. - Error-reply amplification (verified clean). Every polite per-frame
reply charges the per-connection budget and exhausts to one
4006close; relay error branches drop the lifecycle gate before re-acquiring it in the refusal reply; teardown writes are deadline-bounded and first-wins. - Log content and volume (verified clean, one disposition). No
hot-path log includes raw frame bodies, payloads, or credentials; parse
violations log decoder errors without content; upgrade and metrics
rejections use per-source quiet periods. The heartbeat
persistence-failure warnings fire inside the per-player heartbeat
throttle (default 30 s cadence); the documented
Duration::ZERO"update every message" mode combined with a persistently failing database would warn per frame. This is accepted: the mode is explicit operator configuration, the default cadence bounds the warnings, and the failure itself is loud by design. The subscriber insrc/logging.rsis a cold path — filter selection, JSON/text formatting, and a rolling file appender that fails open to stdout (pinned byfile_appender_init_failure_falls_back_without_panicking, guard deliberately leaked for process lifetime) — and carries no payload content, so the call-site review above covers the family.
With this review, the error and logging paths under pressure family has recorded dispositions. The resource-and-input-safety slice closes with one carry-over: its configuration-validation family is half dispositioned — reload consistency carries the 2026-10-01 allowlist/key reload review, while default coherence and validation breadth (malformed documents, env-override interactions) remain with the Config and reload coverage row below.
C1 client and deployment boundaries review (2026-10-04)¶
At a0af63b1 (main after #739), reviewed the C1 slice's client and
deployment boundary families: the four shipped reference clients
(clients/browser, clients/native, clients/fortress,
clients/fortress-wasm) against the reconnect/report/fallback/negotiation
contract, and the plain/TLS listener boundary with the optional features.
Four parallel audits; one reproduced defect class fixed; every family
carries a disposition.
- Reference-client downgrade abort (fixed, ARM-C033). The server
refuses an unsupported requested
game_data_formatwith a budget-chargedErrorenqueued BEFOREAuthenticated(pinned wire order, servertests/e2e_tests.rs) and downgrades the session to JSON. Both the native and the browser reference client kept a first-frame gate that accepted onlyAuthenticated/AuthenticationError, so an opaque request against a default (knob-off) deployment aborted a viable, contract-conformant session: native exit 2expected Authenticated, got Error(reproduced live against a default-config server), browser the same fatal at the equivalent gate. The intended pre-roomProtocolInfo.game_data_formatsdiagnostic was unreachable in both clients for this ordering. Both clients now consume exactly one downgrade notice, adopt JSON for every post-handshake wire decision (send shape, inbound classifier, format gate), emit a non-fatal notice event, and stay fatal for repeat notices, other error codes, and authentication refusals. Native pins:handshake_downgrade_error_adopts_json_and_continues,handshake_stays_fatal_for_other_frames_and_repeat_notices. Browser pin: theadvanceAuthenticateHandshakeblock inorchestrator.test.ts. - Native downgrade live cells (session 362, #741 item 1).
unsupported_opaque_requests_downgrade_to_json_and_relay_between_reference_clientsruns rkyv and protobuf requests against a real server with default encoding knobs. Each cell runs two native client processes. Both must consume one downgrade notice beforeAuthenticated, negotiate v3, reach the shared success barrier, and exit successfully. Each peer must receive exactly the other's JSON payload with the sender UUID and v3 delivery stamps. This extends the ARM-C033 unit controls to the handshake, format adoption, relay, and process completion paths. A temporary mutation that retained the refused encoding failed before room creation. The restored client passed both cells. Restore exercisability remains open in #741. The staged changelog check rejected this test-only work (#797). The checker, hook, and CI dependency detector now exempt the test directories under the four known client roots. Literal roots keep nested runtimesrc/testspaths outside the exemption. Data-driven controls cover both classes and mixed test/runtime changes. No runtime change or changelog entry is needed. - Native cross-format accountability live cell (session 364, #741 item 2).
The opted-in rkyv interop scenario keeps its native-to-native relay wave,
then starts a native JSON recipient and a raw rkyv WebSocket peer on the
same server. The peer sends one opaque payload and a valid JSON continuation.
The recipient must consume the exact
UnsupportedFormatgap report before its advisory, receive only the continuation with the sender UUID and v3 stamps, meet its normal success criteria, and exit successfully. This exercises the shared native accountability path for opaque refusals. It does not add a protobuf cross-format cell. The follow-up wave has one absolute deadline and leaves the existing healthy wave's deadlines intact. Removing native report validation made the process reject the advisory for lacking a prior causal report. The restored source passed the live cell. - Native Fortress negative control (session 366, #741 item 6). Two native game processes drive 600 active callbacks with at most one admission per callback. They then drain their outbound queues and complete the existing final acknowledgement exchange. The healthy validator must reject both reports for completed-send rate and sends per callback. The control also requires a substantial workload, matching delivery ledgers, zero relay faults, and successful process exits. Active traffic metrics freeze before final drain so drain writes cannot inflate the measured rate. Disabling the shared throughput checks made the live negative cell fail. The restored validator passed both live cells. A mutex isolates their timed workloads under the hosted runner's normal parallel test settings. The final healthy run exposed zero rollback on one peer despite clean delivery and checksum gates (#804). The healthy socket holds gameplay inputs from startup until the local game reaches frame four. Synchronization controls still flow. This prevents a startup backlog from bypassing prediction and forces repair within the existing eight-frame window. The socket releases retained inputs in order, then returns to its normal delivery path. Fortress source is private compatibility-fixture code. The changelog gate now classifies the two literal Fortress source roots as internal (#803). Native and browser reference-client runtime changes still need release notes.
- Native Fortress drain and restart (session 367, #741 item 7 slice).
A Unix-only cell waits for both native peers to publish atomic gameplay
checkpoints. Each checkpoint proves an unfinished game at or beyond frame 120,
rollback repair, matching checksums, bidirectional traffic, and zero relay
faults. The harness verifies both peers are alive before it sends SIGTERM
to the real server with a one-second drain grace.
Both peers must fail within one shared bound. Each failure must include
the shutdown advisory and the server's coded 4000
server_shutdownclose. A successful report, an unrelated error, a panic, or the normal process deadline cannot satisfy this control. The server must exit successfully within the separate drain bound. The harness starts a new server on the same port. Fresh peers with new identities must complete the full healthy workload and delivery gates. This proves service after restart, not restoration of the interrupted game. Suppressing the disconnect abort made the live cell fail on an unrelated send error despite receiving the shutdown advisory. The restored fixture passed the drain cell and both existing live workloads. Process and file guards clean owned resources on assertion failure. Inbound overflow and restore exercisability remain open in #741. This cell does not cover WASM shutdown behavior. - Native Fortress handshake close (session 368, #741 item 7 slice).
A Unix-only cell allows the first typed synchronization reply through the
relay and retains later replies in its existing bounded FIFO. Requests and
other controls still flow. Both peers must publish an atomic checkpoint
after a successful partial synchronization step, with zero gameplay frames
and positive completed traffic in both directions.
The harness then sends SIGTERM to the real server. Both peers must receive
the shutdown advisory and coded 4000
server_shutdownWebSocket close, then exit normally with code one within the shared fault bound. Each peer reports its phase again when it handles the close. The harness checks the same identity, partial synchronization, zero gameplay frames, and bidirectional traffic at the close. An earlier checkpoint cannot substitute for this close-time proof. Disabling the reply gate made the live cell fail because a peer had reachedRunningat the close. The source was restored before final validation. Healthy peers and the WASM fixture keep their normal reply delivery. Inbound overflow, restore exercisability, and WASM fault cells remain open in #741. - Native Fortress inbound overflow (session 369, #741 item 7 slice). Both real peers first demonstrate healthy gameplay, rollback, and matching checksums. The harness stops both game consumers at explicit file barriers while transport polling continues. The creator sends genuine captured Fortress inputs in fresh, contiguous relay envelopes. The queue limit remains 256 frames. The receiver must retain exactly 256 frames, identify the first rejected application sequence and server sequence/epoch, and exit with code one within ten seconds. The sender observes the departure; the server remains live. No healthy report, panic, or deadline expiry is accepted. Replacing the overflow error with silent dropping makes the live cell fail its bounded-exit assertion. The shared native/WASM adapter now returns overflow evidence; both owners propagate the error immediately. Native unit coverage pins the admission boundary with valid codec bytes. Restore exercisability and WASM fault cells remain open in #741.
- WASM Fortress active-game drain (session 370, #741 item 7 slice).
The browser harness waits for a Rust-origin
Runningcheckpoint from each no-thread Godot runtime. Each checkpoint requires confirmed gameplay, rollback, matching checksums, and completed bidirectional relay traffic. It then sends SIGTERM to the real server. Both runtimes must stop within ten seconds with the shutdown advisory and coded 4000server_shutdownclose. The advisory stops new gameplay while transport polling continues until each peer receives its own close. Final reports retain the same identities and unfinished gameplay; healthy completion, unrelated failures, panic, and deadline expiry refuse acceptance. The server must exit normally. Existing healthy and capped admission cells retain their exact delivery-ledger checks. Interrupted games allow in-flight frames at shutdown. Validator controls reject the wrong close code, wrong cause, missing advisory, and vacuous checkpoints. WASM restart, partial-handshake close, inbound overflow, and reference restore exercisability remain open in #741. - WASM Fortress restart (session 371, #741 item 7 slice). The runner first requires the active-game drain proof above. It closes both old browsers and waits for the server to exit normally. It then starts the same binary on the same port. Two fresh no-thread Godot peers must pass every released healthy, runtime-identity, and exact delivery-ledger gate. The harness keeps separate drain and restarted-game artifacts, with the binary, build, port, process, and peer identities in a restart record. Identity controls reject reuse across either peer role. Partial-handshake close, inbound overflow, and restore exercisability remain open in #741.
- WASM Fortress partial-handshake close (session 372, #741 item 7 slice).
The
sync-closecell waits for both Rust runtimes to report an actualSynchronizingevent with nonzero, incomplete progress and bidirectional handshake traffic. The relay releases one sync reply and retains later replies, so a retry backlog cannot finish the handshake in one poll. Each runtime freezes Fortress progression at that point and continues polling the real signaling transport. The harness drains the server and requires both peers to report its advisory and coded 4000 close within ten seconds, with no gameplay or unrelated relay fault. It keeps each pre-close checkpoint and terminal report. Validator controls reject empty or completed handshakes, early gameplay, missing traffic, and wrong close causes. Healthy and drain cells hold startup inputs through local frame four, as the native fixture does, to force a real prediction and correction before requiring rollback evidence (#815). A paired-session control covers smooth delivery with zero rollbacks and the configured correction path. Failure artifacts retain periodic Rust-origin probe metrics and active/sync checkpoints. WASM inbound overflow and reference restore remain in #741. - WASM Fortress inbound overflow (session 375, #741 item 7 slice). The browser cell requires two healthy active-game checkpoints before the fault. Both Rust runtimes freeze Fortress consumption while real Godot transport polling continues. The harness waits for existing writes to settle, then releases a bounded creator replay of genuine captured inputs in fresh relay envelopes. The receiver must stop within ten seconds at the unchanged 256-frame limit. Its terminal report identifies the first rejected application sequence, sender, server sequence, epoch, and retained queue. Exact counter and ledger deltas distinguish accepted frames from the rejected frame. Gameplay cannot progress after the freeze. The server stays live; unrelated errors and healthy completion refuse acceptance. A paired-session control reaches healthy rollback and then fills the real adapter with valid inputs. Moving the capacity rejection past frame 256 makes that control fail. Browser oracle controls reject missing or incorrect fault evidence, ledger deltas, traffic, and continued gameplay. Live producer identity comes from its Rust snapshot; receiver identity comes from its terminal report. Healthy and capped-admission cells retain their existing gates. The controlled restore cells below close the final reference-client gap in #741.
- Native controlled restore (session 376, #741 item 4).
Repeated release-file controls exercise real v3 reconnects. The client
captures the private room token, keeps the player and room identity,
requires token rotation and replay status, and replaces membership,
readiness, delivery cursors, and connection counters from the snapshot.
Old engine callback channels and physical pair state are discarded.
A finalized restore waits for a fresh plan. Logical exchange debt survives,
but current members require fresh sends and receipts. Incumbents reset
exchange evidence when the restored incarnation is announced.
The relay cell sends traffic during two confirmed absences, checks exact
nonzero watermarks, and receives the next stamped packet after each
restore. Missed gameplay never becomes a receipt. A live mesh cell sends
reliable data before restore, then requires fresh reliable and unreliable
exchanges after rebuilding both data channels. The Unix drain cell holds
an already-open replacement socket, starts the real server drain, and
requires a bounded
SERVER_DRAININGrefusal with no restore or success. State-machine controls cover malformed identity/token/replay snapshots, stale membership and readiness, callback replacement, and old exchange evidence. OnlyPLAYER_ALREADY_CONNECTEDretries, within a bounded teardown window and the run deadline. Automatic reconnect and game-state resync remain outside this reference client's scope. Browser and released Fortress fixtures do not gain reconnect initiation. - Opaque over a v2 negotiation (fixed with the same sweep). The native
client validated an opaque request against the requested version only;
the
ProtocolInfoNone-version (v2) arm skipped every format check, and the browser's advertisement gate is version-blind.game_data_formatsis not version-gated on the wire, so a v2-capped deployment advertising rkyv would have carried the clients' attribution-less binary shape over v2, violating the reference clients' own v3-only opaque constraint (#627). Both clients now refuse an opaque request on a sub-v3 negotiation (opaque_request_on_a_v2_negotiation_is_refusednative; thenegotiated < 3guard browser-side). - Docs drift (fixed, ARM-C034).
clients/README.mdstill said the Fortress fixtures pin released client 0.8.0/0.9.0; both fixtures pin=0.13.0since #588. The browser README described the runtime as JSON-only, which stopped being true when opaque negotiation landed. Both corrected. - Sound families (verified, no defect). Browser: delivery reports
(exact gaps, counter deltas, advisory causality,
RelayStats), fallback (plan replacement, ICE loss,PeerTransportStatus), and version/format selection match the server; reconnect initiation is absent by documented scope (clients/browser/README.md), with the inboundReconnected/watermark arms correct at unit level. Native: accountability model, transport fallback (cripple, TURN bad-secret, host-star, Direct rejection), and negotiation happy paths match and are pinned. The initial C1 review found no native reconnect initiation or restore exercisability; session 376 adds the controlled restore cells described above. Automatic recovery remains outside the documented scope. Fortress fixtures: the "without silent loss" invariant is substantiated by executable CI gates on both stacks (cross-peer ledger equality, contiguity, queue-age and checksum gating; both families add an asserted expected-negative control); they are WebSocket-only, so they evidence no ICE-fallback claim. - Deployment boundaries (verified, gaps filed). The listener cannot
half-start: every fallible startup step precedes the bind, the listener
socket exists only inside
bind_tcp_listener, and start logs follow the bind on both paths. TLS cannot be silently enabled (validation refuses atls.enabledconfig on a non-TLS binary, pinned + CI lane) and TLS-on fails closed pre-bind on invalid material; mTLS token binding is pinned e2e over the real binary.legacy-fullmeshis binary-local, spawns a separate unauthenticated plane on port+1 with a loud warning and a collision guard, and cannot be reached from the main router.trace-validationandallocation-trackingare inert dev seams. The ARM-C035 and ARM-C036 coverage gaps are closed (issue #740): the drain→4000 close path over TLS and the failure-after-partial-startup regressions are pinned over the real binary intests/tls_deployment_boundaries_e2e.rs. The Fortress native harness gaps are closed by the live cells recorded above (#741).
The client and deployment boundary slice has recorded dispositions. The reference-client gaps in #741 and deployment gaps in #740 are closed. External-platform acceptance remains separate from this code and live-test evidence.
ARM-C044 — Reference-client error diagnostics exposed room tokens¶
| Field | Record |
|---|---|
| State, severity | Fixed, low |
| Player impact | A malformed server snapshot could copy its reconnect bearer token into native and browser client error output. Unexpected snapshot diagnostics could expose it too. |
| Source and revision | clients/native/src/wire.rs::next_server_message and client.rs text/binary decoder and unexpected-handshake diagnostics, and the browser wire decoder, delivery-class validator, and handshake gates, reviewed at 78ef4936 during session 376 (#819). |
| Invariant | Protocol failures stay fatal without emitting room tokens, raw frames, or invalid field values. |
| Confidence and reproduction | Red-first malformed RoomJoined and Reconnected snapshots contain a valid token and an invalid lifecycle value. The handshake and active-session controls fail when either the raw frame or serde error text reaches the error event. Binary controls use token-bearing metadata keys and encoding values; the browser controls use a token-bearing message tag and delivery class. |
| Disposition | Both text paths share a safe decoder. Errors retain frame bytes, error category, line, and column. Unexpected handshake diagnostics omit snapshot payloads. Browser envelope errors and JavaScript parser excerpts also omit input contents. Binary diagnostics omit untrusted decoder text and metadata keys. Token controls cover native handshake/runtime paths, browser classifiers, delivery validation, and the public browser join flow. |
ARM-C033 — Reference clients aborted the server's JSON downgrade handshake¶
| Field | Record |
|---|---|
| State, severity | Fixed, medium |
| Player impact | On any deployment without the opaque-encoding knobs (the defaults), a reference client requesting rkyv/protobuf could not join at all: the session died at the handshake with expected Authenticated, got Error instead of continuing on the JSON floor the protocol guarantees. |
| Source and revision | clients/native/src/client.rs authenticate first-frame gate and clients/browser/src/page/orchestrator.ts authenticate gate, reviewed at a0af63b1; server side src/websocket/connection.rs (Error enqueued before Authenticated, session downgraded to JSON). |
| Invariant | A refused game_data_format downgrades the session to JSON; a client must consume the pinned notice and continue on the downgraded negotiation. |
| Confidence and reproduction | Native reproduced live: default-config server + --protocol-version 3 --game-data-format rkyv --create-room → exit 2 with expected Authenticated, got Error. Browser identical by code reading (same gate shape, same pinned server order). |
| Disposition | Both clients consume exactly one downgrade notice and adopt JSON for every post-handshake wire decision; repeat notices, other error codes, and auth refusals stay fatal. Pins listed in the review section above. |
ARM-C034 — Client documentation described stale pins and a JSON-only runtime¶
| Field | Record |
|---|---|
| State, severity | Fixed, low |
| Player impact | None on the wire; client authors were told the Fortress fixtures validate SDK 0.8.0/0.9.0 (actual pin =0.13.0 since #588) and that the browser runtime never accepts binary frames (opaque negotiation does). |
| Source and revision | clients/README.md:20,21,58 and clients/browser/README.md binary-frame paragraph, reviewed at a0af63b1. |
| Invariant | Shipped documentation describes the shipped flag surface and pin set. |
| Confidence and reproduction | Direct file comparison against clients/fortress/Cargo.toml / clients/fortress-wasm/Cargo.toml and wire.ts's format-conditional classifiers. |
| Disposition | Both documents corrected in this change. |
ARM-C035 — The drain→4000 close path over TLS has no test¶
| Field | Record |
|---|---|
| State, severity | Fixed, medium (coverage gap closed 2026-10-04, issue #740) |
| Player impact | A regression in the TLS shutdown wiring or rustls close would give wss players abrupt closes without the coded 4000 server_shutdown, undetected by CI; the plain-socket contract is pinned, the TLS path was not. |
| Source and revision | src/main.rs TLS serve/shutdown wiring; plain-only contract test tests/close_code_semantics_e2e.rs; reviewed at a0af63b1. |
| Invariant | The drain choreography (GoingAway advisory, coded 4000, bounded exit) holds identically over TLS. |
| Confidence and reproduction | Direct proof of absence: the mTLS e2e spawns the real TLS binary but never exercises shutdown; no TLS drain test existed. No defect claimed. |
| Disposition | tests/tls_deployment_boundaries_e2e.rs::shutdown_drain_over_tls_advises_then_closes_4000_and_exits_bounded spawns the real TLS binary, seats a v3 client over wss, delivers SIGTERM, and pins the advisory (deadline anchored at the drain, retry_after mirroring the configured grace), the 4000 server_shutdown close, and a clean bounded process exit. Unix-only (the drain trigger is a signal); Windows keeps the plain-socket pin. Red-proven: with the configured 1 s grace replaced by the 30 s default, the deadline/retry-after assertions fail loudly. |
ARM-C036 — Failure after partial startup has no regression coverage¶
| Field | Record |
|---|---|
| State, severity | Fixed, low (coverage gap closed 2026-10-04, issue #740; invariant verified holding at a0af63b1) |
| Player impact | None today; a future regression could ship a half-started or falsely-announced server unnoticed. |
| Source and revision | src/main.rs startup order and src/websocket/routes.rs::bind_tcp_listener, reviewed at a0af63b1. |
| Invariant | A fallible startup step after background-task spawn (bind conflict, invalid TLS material) must abort with no listener and no "Server started" announcement. Holds by construction: every ? precedes the bind and logs follow it. |
| Confidence and reproduction | Direct proof of absence: no EADDRINUSE / invalid-PEM / log-ordering test existed. |
| Disposition | Two regressions in tests/tls_deployment_boundaries_e2e.rs pin the invariant (unix-only; Windows keeps the in-process order verification): port_bind_conflict_exits_nonzero_without_start_announcement (an occupied port must exit non-zero, attribute the abort to the address-in-use error, and announce no start) and invalid_tls_pem_exits_nonzero_without_announcement_or_listener (invalid PEM must exit non-zero, attribute the abort to the TLS material, announce nothing, and leave no listener reachable on the port). |
C1 config and reload coverage review (2026-10-04)¶
At a479d3a4 (main), reviewed the last two open families of the Config and
reload coverage row: default coherence and validation breadth (malformed
documents, env-override interactions). Three parallel audits (defaults
coherence, malformed documents, env-override interactions), every candidate
defect reproduced against the real binary before the fix, five defect classes
fixed (ARM-C038..ARM-C042 below), one default-coherence defect fixed
(ARM-C043), and every family carries a disposition.
- Non-object JSON source silent revert (fixed, ARM-C038). A JSON source
whose root was not an object replaced the whole merged document
(
merge_valuescatch-all), so a truncated write leavingnullin a mid-precedenceconfig.jsonsilently discarded every lower-priority source and booted on compiled defaults plus whatever the higher sources set — exit 0, process healthy (reproduced live:nullcwd file + inline{"port":1234}→ port 1234, all other knobs default). Position-dependent: the samenullin the highest source hard-errored. Now any present source with a non-object root is a hard error naming the source and the root kind, at every precedence position (non_object_json_source_is_a_hard_error_naming_the_source). logging.levelsilent revert (fixed, ARM-C039). The customLoggingConfigdeserializer downgraded an unrecognized level string to a stderr note and a wrong-typed value (number/bool/object) to a silentNone— the only knob where a present-but-invalid value did not failload(), contradicting the loader contract and the siblinglogging.rotationtreatment (reproduced live:level:7→ exit 0,level: null, no diagnostic). The field now parses throughLogLevel's own strict deserializer; case/whitespace tolerance and thewarning/erraliases survive, unknown strings and non-string values are hard errors (invalid_log_level_string_is_a_hard_error,non_string_log_level_is_a_type_error,log_level_aliases_and_case_are_still_accepted).- Stat-failing config file treated as absent (fixed, ARM-C040).
read_file_sourcegated onPath::exists(), which reportsfalsefor a broken symlink, symlink loop, or unsearchable parent, so a configured file that could not be stat'ed was silently skipped in favor of lower-priority sources (reproduced live:SIGNAL_FISH_CONFIG_PATHat a symlink loop → defaults, exit 0; a dangling symlink behaved identically). Now the source's directory entry is checked without following the final symlink (symlink_metadata):NotFoundthere is the one tolerated outcome, and every present-but-unreadable shape — dangling symlink, symlink loop, permission errors — is a hard error naming the path (config_file_that_fails_stat_is_a_hard_error_naming_the_path, covering both symlink shapes). - Env override key depth overflow (fixed, ARM-C041).
set_nested_valueand the resultingValue's recursive drop recursed once per__segment with no cap;std::envpermits far deeper keys than any config path. Reproduced live: a 40k-segmentSIGNAL_FISH__A__A__…variable aborted the process at startup with a stack overflow (SIGABRT). Overrides beyond 16 segments (MAX_ENV_OVERRIDE_DEPTH; the deepest canonical path is 4) now fail with a named error (env_override_deeper_than_the_segment_cap_is_a_hard_error). The same class was already closed on the document path: serde_json's 128-level recursion limit bounds every JSON source and env JSON value, now pinned for the config path (deeply_nested_config_source_is_a_parse_error_not_a_crash). - Order-dependent case-variant duplicate env overrides (fixed, ARM-C042).
Two case variants of one override name (
SIGNAL_FISH__PORTandSIGNAL_FISH__port) map to one knob; resolution was first-in-iteration (unspecifiedstd::env::vars()order) behind a misleading "canonical and legacy" warning (reproduced live). Same-class duplicates now hard-error naming both variables; true canonical-vs-legacy alias pairs keep their pinned order-independent canonical-wins resolution (duplicate_case_variant_env_overrides_are_a_hard_error_naming_both_vars; existingcanonical_app_access_env_keys_win_over_legacy_keys_in_either_order/canonical_app_list_env_wins_over_legacy_list_in_either_orderre-run green). - Default per-IP budget equaled one default roster (fixed, ARM-C043).
default_max_connections_per_ip(24) equaled exactly one fully seated default room (8 players + auto-derived 2× spectators = 16), leaving zero per-IP slack for reconnect churn behind one NAT, while its own comment claimed 16-player headroom (arithmetically 48 seats). The default is 64 with a corrected comment, pinned by a data-driven coherence test (default_per_ip_budget_admits_a_full_nat_roster_with_churn_slack); the config-reference drift guard and docs were updated with it.
Clean dispositions:
- Defaults vs guards: with the documented-by-design metrics-credential
gate satisfied,
Config::default()passes every numeric, cross-field, protocol-window, transport, TURN, session, metrics, logging, and rate guard (guard-by-guard trace recorded in the session notes); enum defaults round-trip through the loader's serialize→merge→deserialize pipeline. - Malformed documents: unparsable JSON, wrong types at any depth (file
side now pinned beside the pinned env side:
wrong_typed_file_value_is_a_hard_error_naming_the_knob), u8/u64 overflow, floats on integer knobs, invalid UTF-8, BOM, trailing garbage, directory-at-path, and permission errors all hard-error with source or knob attribution. Duplicate JSON keys are last-wins by serde semantics; legacy normalization runs post-dedup and cannot be confused by them. - Unknown keys: the deny-strict-security / tolerate-elsewhere split is
deliberate, structurally pinned, and operator-documented; removed-key
tolerance is pinned. The one credential-bearing structure outside the
security subtree (
session.ice_serversentries) tolerates unknown keys by the same documented policy, and a missing TURN username/credential is warned at validation with the entry index. - Env interactions: precedence (env over every JSON source, source order
1→6 as documented), scalar/JSON/comma-array value parsing, empty-value
refusals, secret redaction (
REDACTED_SECRET+ legacy discard paths hard- error or strip on every intake path), the app-registry merge (append, not replace, with downstream duplicate rejection), and the connect-token inline-vs-path conflict are all pinned or dispositioned. The SIGHUP reload re-runs the identical load pipeline (sources, env, folds, validation), so startup and reload accept/reject identically. - Docs vs code: the ~70-row configuration reference matches the code
defaults; the audit's three documentation drift findings (run-modes rows
whose "exact command" fails under the fail-closed metrics gate, the
development snippet missing
require_metrics_auth, the README Docker trial missing the CORS override its browser-console step needs) are fixed in this change.
ARM-C038 — A non-object JSON config source silently discarded lower-priority sources¶
| Field | Record |
|---|---|
| State, severity | Fixed, medium |
| Player impact | A corrupted/truncated write leaving null (or a hand edit leaving an array/scalar) in a mid-precedence config.json reverted every lower-priority operator setting (allowlists, caps, timeouts) to compiled defaults while the process appeared healthy; under the documented README posture where env carries the security knobs, the silent revert was fully invisible. |
| Source and revision | src/config/loader.rs parse_json_document (accepted any JSON root) and merge_values catch-all (non-object source replaced the accumulated document), reviewed at a479d3a4. |
| Invariant | A config source that is present but invalid is a hard error naming the source, at any precedence position — never a silent substitution of defaults. |
| Confidence and reproduction | Reproduced live at a479d3a4: cwd config.json containing null + SIGNAL_FISH_CONFIG_JSON='{"port":1234}' → exit 0, port 1234, every other knob at compiled defaults. |
| Disposition | parse_json_document requires an object root and errors naming the source and root kind (non_object_json_source_is_a_hard_error_naming_the_source); the loader contract doc lists the non-object case explicitly. |
ARM-C039 — An invalid logging.level silently reverted to the default¶
| Field | Record |
|---|---|
| State, severity | Fixed, low |
| Player impact | None directly; an operator typo ("warng") or wrong-typed value (7) started the server on the default log level — on a deploy expecting trace or error, diagnostics were silently thinner than configured. |
| Source and revision | src/config/logging.rs custom LoggingConfig deserializer (Option<serde_json::Value> + lenient coercion), reviewed at a479d3a4. |
| Invariant | A present-but-invalid config value fails load() — the same treatment every other knob gets — instead of silently reverting. |
| Confidence and reproduction | Reproduced live at a479d3a4: SIGNAL_FISH_CONFIG_JSON='{"logging":{"level":7}}' → exit 0, level: null, zero diagnostics; "warng" → exit 0 with only an unspecific stderr note. |
| Disposition | The field is Option<LogLevel>, parsed by LogLevel's strict deserializer (case/whitespace-tolerant, warning/err aliases kept); unknown strings and non-string values are hard errors. Pins: invalid_log_level_string_is_a_hard_error, non_string_log_level_is_a_type_error, log_level_aliases_and_case_are_still_accepted. |
ARM-C040 — A stat-failing config file was silently treated as absent¶
| Field | Record |
|---|---|
| State, severity | Fixed, low |
| Player impact | None directly; a deployment whose SIGNAL_FISH_CONFIG_PATH (or cwd config.json) resolved to a broken symlink or symlink loop silently ran on lower-priority sources — the exact revert-on-provisioning-failure shape the app-registry fold is fail-closed against. |
| Source and revision | src/config/loader.rs read_file_source Path::exists() gate, reviewed at a479d3a4. |
| Invariant | NotFound on the source's directory entry (checked via symlink_metadata, not following the final symlink) is the one tolerated file outcome; every present-but-unreadable shape — dangling symlink, symlink loop, permission errors — is a hard error naming the path. |
| Confidence and reproduction | Reproduced live at a479d3a4: SIGNAL_FISH_CONFIG_PATH at a self-referential symlink → defaults, exit 0, no log; a dangling symlink behaved identically (verified during adversarial review of the first fix, which only covered the loop). |
| Disposition | The exists() gate is replaced by symlink_metadata (entry existence without following the final symlink); NotFound → Ok(None), everything else the path-naming hard error (config_file_that_fails_stat_is_a_hard_error_naming_the_path, pinned for both the loop and the dangling-symlink shapes). |
ARM-C041 — A deep env override key crashed the process via stack overflow¶
| Field | Record |
|---|---|
| State, severity | Fixed, medium (robustness; input requires control of the process environment or a corrupted launcher script) |
| Player impact | A hostile, corrupted, or buggy launcher environment variable (SIGNAL_FISH__A__A__…) aborted the whole server at startup or SIGHUP — every room's signaling dropped — with no diagnostic beyond the runtime's stack-overflow abort. |
| Source and revision | src/config/loader.rs set_nested_value recursion and the recursive Drop of the built Value; no depth cap on __-separated override keys, reviewed at a479d3a4. |
| Invariant | Recursion over external input is bounded by a named limit (the same class as the wire-decoder depth walls; cf. ARM-C030). |
| Confidence and reproduction | Reproduced live at a479d3a4: 40k-segment key → fatal runtime error: stack overflow, SIGABRT (exit 134), on the main thread at startup. |
| Disposition | Overrides beyond MAX_ENV_OVERRIDE_DEPTH (16; deepest canonical path is 4) fail with a named error (env_override_deeper_than_the_segment_cap_is_a_hard_error). The document path was already bounded by serde_json's 128-level recursion limit, now pinned for the config path (deeply_nested_config_source_is_a_parse_error_not_a_crash). |
ARM-C042 — Case-variant duplicate env overrides resolved by unspecified iteration order¶
| Field | Record |
|---|---|
| State, severity | Fixed, low |
| Player impact | None directly; two case variants of one override (SIGNAL_FISH__PORT vs SIGNAL_FISH__port) silently resolved by unspecified std::env::vars() order behind a misleading "canonical and legacy" warning, making the effective config launcher-order-dependent. |
| Source and revision | src/config/loader.rs env-override Occupied entry arm, reviewed at a479d3a4. |
| Invariant | An ambiguous override is refused deterministically; the canonical-vs-legacy alias warning is factually accurate and keeps its order-independent resolution. |
| Confidence and reproduction | Reproduced live at a479d3a4: both variants set → legacy-conflict warning (false) and first-iteration winner. |
| Disposition | Same-class duplicates hard-error naming both variables and the knob path; cross-class alias pairs keep canonical-wins with an accurate warning including both names (duplicate_case_variant_env_overrides_are_a_hard_error_naming_both_vars; existing either-order alias pins re-run green). |
ARM-C043 — The default per-IP connection budget equaled exactly one default room roster¶
| Field | Record |
|---|---|
| State, severity | Fixed, low (default coherence) |
| Player impact | A fully seated default room behind one NAT (8 players + auto-derived 16 spectators = 24 registrations, every seat through the per-IP limiter) consumed the default security.max_connections_per_ip exactly, so the first reconnect churn or extra tab from that NAT was refused IpLimitExceeded; the default's own comment promised 16-player headroom (arithmetically 48 seats) that did not exist. |
| Source and revision | src/config/defaults.rs default_max_connections_per_ip (24) vs src/server/room_service.rs auto spectator capacity (2× player ceiling) and the connection_manager per-IP registration gate, reviewed at a479d3a4. |
| Invariant | Compiled defaults compose: the documented NAT/LAN use case (16 players + 32 auto spectators) fits inside the default per-IP budget with reconnect churn headroom. |
| Confidence and reproduction | Static roster arithmetic over the registration path (all registrations — players, spectators, reconnects — consume a per-IP slot in register_delivery). |
| Disposition | Default raised 24 → 64 with a corrected comment; pinned by default_per_ip_budget_admits_a_full_nat_roster_with_churn_slack (data-driven over the roster formula). The parallel ServerConfig::default() literal in src/server.rs now derives from the same default_max_connections_per_ip() instead of a divergent hardcoded 24. Config-reference table, deployment docs, checklist, and example configs updated to the new default. |
ARM-C045 — Weak tenant keys accepted forged connect tokens¶
At 49731ecb (2026-10-08), the connect-token key loader accepted low-order
Ed25519 public keys. Permissive verification accepted R = identity, S = 0
under the identity key for arbitrary tenant claims without a signing key.
This requires an operator-configured weak key; generated signing keys are
not affected. See #821.
- Red evidence:
weak_verification_keys_are_refused_in_every_config_encodingaccepted the identity key, andidentity_key_cannot_accept_a_signature_forged_without_a_private_keyaccepted the forged tenant claim against unchanged production code. - Fix: the shared key constructor refuses low-order points and credential
verification uses
verify_strict. Configuration validation, startup, direct key installation, and SIGHUP all use that constructor. - Controls: identity and order-two keys across all supported base64 encodings; arbitrary forged claims; startup validation; SIGHUP rejection preserving the running allowlist, key, and enforcement posture. Existing RFC key, valid-token, expiry, TTL, app-ID, and rotation controls remain.
- Scope: this closes weak-key admission. The claim-boundary review below closes the remaining connect-token audit. The weak-key finding is ARM-C045; ARM-C044 identifies the earlier client diagnostic finding.
ARM-C046 — Signed connect-token arrays bypassed the object contract¶
At 4c26fb23 (2026-10-08), a correctly signed JSON array
["app",1300,"n"] passed verification. Derived struct deserialization
accepted positional claims despite the documented object format. This needs
a trusted signing key; it is not a signature bypass. See
#823.
- Red evidence:
signed_claims_require_a_json_objectfailed on unchanged production because the signed array was accepted. - Fix: after signature verification, require an object before typed parsing. Valid JSON whitespace and extension fields remain accepted.
- Controls: signed arrays and scalar shapes; duplicate claims, including
escaped field names; non-integer and out-of-range expiry; malformed and
trailing JSON; signature-before-claims ordering; exact decoded app identity;
Unicode nonce byte limits; replay and clock changes. All 21 token unit
controls pass. The real-socket
signed_claim_array_is_refused_without_committing_admissioncontrol proves refusal withCONNECT_TOKEN_INVALIDand valid retry on the same socket.
C1 connect-token claim-boundary review (2026-10-08)¶
Reviewed src/security/connect_token.rs, the key install and verification
wrappers in src/server.rs, and admission in src/websocket/connection.rs.
Encoding and total length precede strict signature verification. Object
shape, required typed fields, duplicate known fields, empty identity, and
nonce byte length precede expiry and remaining validity. App binding compares
the decoded identity exactly; no Unicode normalization or case folding occurs.
Existing controls cover token size, missing fields, empty values, unknown
extensions, signature tampering, expiry, TTL, key rotation, and removal.
The new raw-payload controls close the claim ambiguity gaps above.
Admission resolves and charges the app budget before credential checks, then verifies before capability negotiation or handshake state commit. Every token refusal uses the charged error-reply path and remains retryable. Existing real-socket controls cover app mismatch, missing key, expiry, invalid signature, and global/per-app enforcement; the new shape control covers malformed signed claims on that same path. Errors contain no claim or credential bytes.
Intentional exclusions: verification uses the current key snapshot and server wall clock; clock changes can change validity. The TTL ceiling bounds remaining validity at admission, not lifetime since minting. No issue-time claim exists. Replay is accepted until expiry, nonce consumption belongs to the operator's edge, and expiry does not revoke an admitted session. Unknown extension fields are tolerated. These policies are documented; no further claim defect was found. Independent adversarial review found no remaining issue in this slice.
C1 protocol subsystem review (2026-10-05)¶
At 8f80cc6e (main after #746), reviewed the protocol coverage row across
src/protocol/** and src/trace_validation.rs against the row invariant
"V2/V3 decoding, wire bytes, and delivery class match contract", with the
named missing cases (malformed/deep frames, mixed format boundaries) and
the fuzz target fuzz/fuzz_targets/decode_protocol.rs. No defect was
found; every hazard family carries a recorded disposition and the two
previously undocumented decode behaviors are now pinned.
- Delivery-class decode carries explicitly (verified).
class/keyareOptionfields decoded throughdeserialize_present_optional: omission isNone, an explicitnullfails decode, and no present value is ever defaulted at decode. TheNone → reliablefloor is a relay-layer rule exactly where the documented "omitted class means reliable" lives, binary frames are hardwired reliable by contract, the v2/v3 gate admits exactly the documented pairings, and theINVALID_DELIVERY_CLASSbefore size/seat ordering matches the documented validation order (all previously pinned). - Strict v3 binary envelope (verified, previously pinned). The reference decoder rejects duplicate/unknown fields, trailing bytes, non-UUID senders, and out-of-range stamps without truncation; boundary stamps decode verbatim; per-frame encoding is negotiated, never sender-controlled; unsupported formats fail with the exact gap plus the throttled advisory (ARM-C032 cadence).
- Depth and size limits are symmetric across paths (verified). The
JSON command path leans on serde_json's 128-level recursion limit
(documented at the scanner constant), the MessagePack ingress and
fallback conversion run the shared iterative 128-level scanner first
(ARM-C030), frame and payload caps precede decode on both lanes, and
the deep-nesting probes pin clean
Errbehavior on both formats (tests/protocol_fuzz_hardening.rs). - Enum and tag handling is fail-closed (verified). Unknown class,
encoding, transport, topology,
type, or error tokens and explicitnullmetadata fail decode withINVALID_INPUTand the connection kept open;ErrorCodehas no catch-all variant, so unknown codes are never silently coerced. - Duplicate members have pinned precedence (new pins). The plain
JSON lane rejects duplicate members at every typed envelope level —
tag, content member, and content fields (
duplicate field …decode error) — matching the token-bound lane's frame-wide pre-verification rejection; only opaque payload values collapse, deterministically last-wins, which is the only representable outcome once a never-inspected payload is decoded into a JSON value map (the MessagePack path and the Json-in-binary fallback decodedecode_binary_to_jsonshare the sameValuesemantics, so they collapse the same way). Pinned byjson_duplicate_members_have_pinned_precedence. - Numeric literals have pinned fidelity (new pins). Integer literals
inside the
i64/u64range relay exactly; literals outside it take serde_json's f64 path and relay as the nearest f64's shortest text (2^64 + 1 drifts to f64-exact 2^64 — the documented, bounded float approximation class, not a new defect; integer literals outside thei64/u64range are the pinned exception to the unqualified "relayed verbatim" payload guarantee, and clients needing exact large integers have the signed lane's interoperable range as the normative contract); literals beyond the f64 range fail decode cleanly instead of relaying an infinity ornull;-0re-renders value-equal in float form. The exact-text pins make any serde_json number-handling change a loud wire-contract event. Pinned byjson_integer_literals_relay_within_the_pinned_fidelity_contractandjson_out_of_range_float_literals_fail_decode_cleanlyintests/v3_wire_properties.rs. trace_validation.rsis a recorder, not a validator (verified). No wire frame passes through it, so no decode bypass exists; its fail-closed divergence labeling and queue-close races are pinned in-module, and its two documented silent drops stay documented.
The Protocol coverage row moves to reviewed: wire bytes, delivery class, and depth/size behavior match the contract on every decode path — with the pinned out-of-i64/u64 integer-literal exception recorded above — and the remaining numeric/member behaviors are pinned with recorded dispositions.
Coverage ledger¶
All rows were inventoried at b24b5e13; the review state of each row is
recorded in its State column below. Paths identify the review seam;
listed tests, models, and fuzz targets are leads, not completed reviews.
Check default, tls, legacy-fullmesh, trace-validation, and
allocation-tracking feature combinations where applicable (Cargo.toml).
The library modules exported by src/lib.rs, the binary, and shipped reference
clients are covered below.
Public entry points to check: src/main.rs builds the root listener, including
/v2 routes, /v3/client-config, /health, /readyz, and metrics endpoints.
src/websocket/routes.rs builds /ws, /client-config, /health, /readyz,
/metrics, /metrics/prom, /v3/ws, and /v3/client-config; its router is
nested under /v2 by the binary. The binary also has a legacy listener path.
The library API surface is the public modules exported from src/lib.rs:
auth, config, coordination, database, distributed, logging,
metrics, protocol, rate_limit, reconnection, retry, security,
trace_validation, server, and websocket. Each row's review state is
recorded in the coverage table below.
Feature coverage also starts unreviewed. Exercise default=[], tls,
legacy-fullmesh, tls,legacy-fullmesh, and --all-features. The legacy path
is for local interop and has a separate security posture. trace-validation
is an internal verification seam; allocation-tracking is a development
benchmark seam. Check those two in the relevant tests and all-feature build;
neither is a deployed capacity preset.
| Subsystem and paths | Invariant to check | Existing evidence lead | Missing cases / next check | State |
|---|---|---|---|---|
Startup and CLI: src/main.rs, src/lib.rs |
Startup rejects bad config; startup failure leaves no listener | tests/config_and_endpoints_tests.rs, tests/tls_deployment_boundaries_e2e.rs; C1 client and deployment boundaries review above |
Startup order verified: every fallible step precedes the bind and start logs follow it, so no half-started server is reachable; the failure-after-partial-startup regressions (bind conflict, invalid PEM post-spawn) are pinned over the real binary (ARM-C036 closed) | Reviewed |
Config and reload: src/config/** |
Defaults, validation, and reload preserve one coherent policy | tests/config_and_endpoints_tests.rs, tests/config_validation_coverage_scan.rs; C1 allowlist and key reload boundary review and C1 config and reload coverage review above |
SIGHUP key/allowlist swap order and invalid reload reviewed and pinned; default coherence verified guard-by-guard against Config::default(); malformed-document and env-override breadth reviewed with five fixed defect classes (ARM-C038..ARM-C042) and the per-IP default-coherence fix (ARM-C043), all red-proven and pinned |
Reviewed |
Authentication: src/auth/**, src/rate_limit.rs |
Unauthorized traffic cannot enter a room; limits count refusals | tests/auth_integration_tests.rs, formal/tla/RateLimitWindow.tla; C1 authentication admission boundary and rate-limit rejection accounting reviews above |
Flood posture, budget-before-credential ordering, refusal closes, the absolute activity-immune auth deadline (pre_handshake_activity_does_not_extend_the_auth_deadline), concurrent ceiling conservation (concurrent_handshakes_conserve_the_app_ceiling_and_count_every_rejection), and every refusal path's exact-once charge/counter pairing (the drain-window creation refusal's deliberate budget-free shape is now pinned) are reviewed and pinned |
Reviewed |
Security: src/security/**, src/websocket/token_binding.rs |
Token, origin, TLS, and TURN credential checks fail closed | tests/mtls_token_binding_e2e.rs, tests/tls_deployment_boundaries_e2e.rs, fuzz/fuzz_targets/fuzz_reconnect_tokens.rs; C1 token rotation boundary and client/deployment boundaries reviews above |
Rotation ordering and concurrent-claim refusals are reviewed and pinned; the TLS-variant posture is verified (silent enable impossible, cert/key fail closed pre-bind, mTLS binding pinned e2e over the real binary) and the drain close path over TLS is pinned over the real binary (ARM-C035 closed); weak-key admission (ARM-C045) and signed object shape (ARM-C046) are fixed; connect-token claim boundaries are reviewed and pinned with the explicit policies above | Reviewed |
Protocol: src/protocol/**, src/trace_validation.rs |
V2/V3 decoding, wire bytes, and delivery class match contract | tests/v2_wire_golden.rs, tests/v3_wire_properties.rs, fuzz/fuzz_targets/decode_protocol.rs; C1 protocol subsystem review above |
Malformed/deep frames, mixed format boundaries, delivery-class carry, enum/tag fail-closed behavior, duplicate-member precedence, and numeric-literal fidelity are reviewed and pinned (or previously pinned); the recorder-only trace_validation.rs has no decode seam |
Reviewed |
Room and player storage: src/database/** |
Membership and room limits stay atomic and app isolated | tests/integration_tests.rs, tests/model_based_state_machines.rs; C1 admission-limit review above |
Other adapters, rollback, and leave/disconnect races remain | Unreviewed |
Room lifecycle and moderation: src/server/room_service.rs, moderation.rs, spectator_service.rs, spectator_handlers.rs |
Join, leave, kick, ban, spectator state and ownership agree | tests/lobby_integration_tests.rs, src/server/room_service_tests.rs; C1 admission-limit, leave/disconnect ordering, and identity-slice completion reviews above |
ARM-C001–C004 fixed in spectator and room-code seams; identity cases (concurrent limits, join-only, leave/disconnect, spectator transitions, kick/ban races, application isolation) reviewed and pinned or derived; storage-fault interleavings on other adapters remain | Partially reviewed |
Readiness and gameplay: src/server/ready_state.rs, authority.rs, session_policy.rs, signaling.rs |
Membership and transport changes invalidate stale plans/readiness | tests/v3_session_plan_e2e.rs, formal/tla/SignalFishSession.tla; C1 gameplay-transitions review above |
Start/leave, authority loss, v2/v3 negotiation, capability intersections, stale reports, downgrade reconnects, and publication order are reviewed and pinned (including the spectator start-authorization coupling); shared src/server.rs state seams remain |
Partially reviewed |
Relay routing: src/server/game_data.rs, message_router.rs, messaging.rs, relay_policy.rs |
Each permitted message reaches only valid peers with correct sequence/class | tests/v3_game_data_sequencing_e2e.rs, tests/mixed_encoding_relay_e2e.rs; C1 cross-room stall fairness, mixed encoding/unsupported conversion, and permitted volatile loss reviews above |
Slow-recipient isolation, cross-room stall fairness, the mixed encoding/unsupported conversion matrix (direct cohorts, lossless fallback, opaque refusal with exact gap plus advisory, pre-v3 advisory-only wire), and real-socket volatile eviction with exact reports plus a non-zero per-connection dropped_for_you (flooded_nonreading_recipient_observes_exact_volatile_gaps_and_dropped_for_you) are reviewed and pinned |
Reviewed |
Coordination and queues: src/coordination/**, src/distributed.rs; the in-memory coordinator seams in src/server.rs |
Transaction and queue failure is explicit; one room cannot strand another | tests/relay_backpressure_e2e.rs, formal/tla/RoomMessageTransaction.tla; C1 room-event duplicate-delivery, latest coalescing keys/generations, and transaction reservation/commit cancellation/panic reviews above |
Lane job exactly-once and no lease re-run are dispositioned and pinned (interleaved_awaits_deliver_each_lobby_broadcast_exactly_once); cross-epoch gap ranges stay distinct per epoch (cross_epoch_gaps_of_one_sender_stay_distinct_ranges); latest key composition, generation shielding, supersession, saturation, and counter conservation are reviewed and pinned; cancellation/panic at reservation and commit are reviewed, the silent panic-accounting class is fixed, and all three fixed seams are pinned (panicking_commit_hook_releases_and_accounts_every_reservation, panicking_phase_callback_accounts_remaining_frames_and_never_delivers_phase_one, panicking_broadcast_replay_hook_releases_and_accounts_every_reservation); other src/server.rs state seams remain with their own rows |
Reviewed |
WebSocket ingress and egress: src/websocket/** |
Bounded frames, priority control, close and drain semantics hold | tests/transport_frame_limits_e2e.rs, tests/slow_consumer_no_cascade_e2e.rs, tests/tls_deployment_boundaries_e2e.rs; C1 parser-boundary, error/logging pressure, and client/deployment boundaries reviews above (ARM-C030 bounded-depth MessagePack decode; ARM-C031/ARM-C032 log hardenings) |
Slow reader, batching age; the drain→4000 close path over TLS is pinned over the real binary (ARM-C035 closed) | Partially reviewed |
Reconnect and retry: src/reconnection.rs, src/retry.rs, src/server/reconnection_service.rs |
Claims have one owner; replay and stale routes cannot leak or misroute | tests/reconnect_window_races_e2e.rs, formal/tla/ReconnectionClaimLifecycle.tla; C1 reaper-ordering, claim-expiry, failed-restore, token-rotation, reconnect epoch/sequence, and inactive-record/claim-retention reviews above |
Simultaneous claim, expiry during claim, failed restore/retry, rotation boundaries, reconnect epoch/sequence transitions (including cross-epoch gap accounting), the pre-issued-token teardown discards (layered, hypothesis falsified), claim/pending-detach retention lifecycles, and the owned-task supervisor-panic strand (ARM-C037 fixed with a red-proven pin) are reviewed and pinned; src/retry.rs backoff seams and multi-failure detach accounting on failing backends remain |
Partially reviewed |
Maintenance and deadlines: src/server/maintenance.rs, heartbeat.rs, dashboard_cache.rs, src/deadline.rs |
Expiry and cleanup are bounded; live state survives sweeps | formal/tla/RoomLifecycleGC.tla, tests/clock_source_scan.rs; C1 maintenance and deadlines expiry boundary review above |
Expiry boundaries and clock sources are reviewed and pinned (reaper pair boundary, zero-timeout disable, monotonic windows, wall-clock-step immunity, overflow); churn growth and dashboard cost remain measurement work | Partially reviewed |
Metrics and logging: src/metrics.rs, src/logging.rs, src/websocket/metrics.rs, prometheus.rs |
Counters report outcomes; labels and logs stay bounded and safe | tests/config_and_endpoints_tests.rs, tests/websocket_test_helpers/prometheus_scrape.rs; C1 metrics label cardinality and error/logging pressure reviews above (ARM-C031/ARM-C032 fixed the rejected-ID log forgery and the unthrottled undeliverable-relay warning) |
Cardinality is bounded with pinned lifecycles; hot-path log content, amplification, and throttle cadences are reviewed and pinned or dispositioned | Reviewed |
Admin and shutdown: src/server/admin.rs, shutdown.rs, connection_manager.rs |
Drain closes all owned tasks and reports queued work accurately | tests/close_code_semantics_e2e.rs, formal/tla/ConnectionTeardown.tla; C1 drain/shutdown review above |
The drain choreography, the reconnect-commit fence, close ordering with queued data, and the drain reservation accounting are reviewed, fixed where defective, and pinned; a distinct 4000-close counter remains follow-up observability | Partially reviewed |
Browser client: clients/browser/src/** |
Reconnect, delivery reports, fallback, and negotiation match server | clients/browser/src/page/*.test.ts; C1 client and deployment boundaries review above |
Reports, fallback, and negotiation verified and pinned (plus the ARM-C033 downgrade fix and its pin); reconnect initiation absent by documented scope with inbound arms unit-pinned; a live browser accountability cell, loss→recovery transitions, and browser-as-host remain | Partially reviewed |
Native client: clients/native/src/** |
Same client contract across native sockets | clients/native/tests/interop_e2e.rs; C1 client and deployment boundaries review above |
Accountability model, fallback (cripple/TURN/host-star/Direct rejection), and negotiation verified and pinned (plus the ARM-C033 downgrade fix, the v2-opaque guard, and their pins); native downgrade and cross-format advisory consumption now have live cells; the MessagePack cohort is pinned; controlled v3 JSON relay restores, rotated-token reuse, fresh WebRTC exchange, and Unix drain refusal have live cells; automatic recovery and game-state resync remain outside shipped scope | Reviewed |
Fortress clients: clients/fortress/src/**, clients/fortress-wasm/src/** |
Reference peers handle relay without silent loss | clients/fortress/tests/multiprocess.rs, clients/fortress-wasm/harness.mjs, both interop workflows; C1 client and deployment boundaries review above |
Silent-loss detection substantiated by executable CI gates on both stacks (contiguity, cross-peer ledger equality, queue-age/checksum gating; both families add an asserted expected-negative control); fixtures are WebSocket-only and evidence no ICE-fallback claim; Unix native drain/restart and partial-handshake close cells prove bounded causal failures; fresh healthy games pass after restart; WASM drain/restart and partial-handshake close cells check bounded causal failures and fresh healthy games on the same port; native and WASM inbound-overflow cells check the real queue boundary and first rejected frame; these released fixtures do not initiate reconnects, while native reference restore is covered above | Partially reviewed |
src/server.rs owns shared server state across the server rows. src/websocket/routes.rs
and src/main.rs own the plain/TLS listener boundary. src/config/coordination.rs
and src/distributed.rs provide local coordination seams; they do not establish
cross-node room replication. External mobile/Steam clients, operated TURN, and
AWS hardware are outside this code inventory and need separate evidence.
Existing measurement evidence and limits¶
tests/websocket_test_helpers/delivery_ledger.rschecks exact receiver, sender, and sequence outcomes or an explicit disconnect. The V3 tests andtests/multiprocess_delivery_e2e.rsprovide real-socket regression leads.formal/tla/**,formal/z3/**, andfuzz/fuzz_targets/**explore related state spaces. None substitutes for an executable regression at a found seam.tests/load_tests.rsruns in-process smoke workloads. The shared-process H2 numbers in Scaling include the generators on the same runner. Neither measures the standalone ARM server ceiling or proves scheduled-send latency under overload.benches/relay_allocations.rs,relay_serialization_allocations.rs, andrelay_serialization_runtime.rsisolate relay costs and preserve exact-wire and delivery checks. They do not include full socket, TLS, kernel, or standalone-server costs. Keep their exact-byte and ledger checks in later comparisons.- #207's allocation profile found 0 queue allocations and 1 fan-out core allocation per relay in its measured layers. Its stored-baseline Criterion deltas had about 25% noise. Neither result is a whole-server ceiling.
- #636's follow-up rejected extra JSON buffer reservation, cache-hit clone changes, and recipient-vector work without new hot-path evidence. Its arm64 microprofile used a 12-core host and 1,024-relay microbench; socket and syscall costs were outside that scope. PR #644's result already reduced mixed-room duplicate JSON body encoding from two passes to one. Do not register that candidate again as pending work. These are historical findings, not measurements from this ledger revision.
Experiment contract¶
Register an immutable record before each measured comparison. Store the record and raw artifacts under a run ID outside the source tree; link them from the finding or campaign issue. A result without the manifest, complete attempts, and outcome ledger is provisional.
| Field | Required value |
|---|---|
| Identity | Run ID, schema version, hypothesis, decision owner, registration time |
| Revisions | Baseline and candidate SHAs, binary/config hashes, features, toolchain |
| Workload | Endpoint, seed, protocol/encoding mix, room/player counts, payload bytes, per-sender rate, delivery classes, warm-up, duration, churn/reconnect schedule |
| Environment | ARM CPU model, kernel, two-CPU affinity and quota, 4-GiB cgroup limit, swap policy, socket limits, TLS path, generator host/resources and network path |
| Primary measure | Scheduled application send to recipient receipt p99; pass at <= 50 ms for relay cells |
| Safety and validity | Exact delivery-class outcomes, all offered work, queue bounds, generator lag, clock error or same-clock method, memory limit, disconnects |
| Comparison | Predeclared run order, all attempts, raw interval/histogram/log artifacts, paired result, uncertainty, accept/reject/defer reason |
Measure idle density, churn, active relay, and mixed loads separately. Never compare unsynchronized host timestamps to the one-way 50-ms target. Increase load only after generator negative controls pass. Keep load generators outside the server's CPU and memory limits. Record unavailable counters and invalid runs; do not omit them. AWS validation remains required for deployment claims.
Next PR contract¶
C1 first slice: identity and membership (#647). COMPLETE (2026-10-01, five slices through 2026-10-04; see the review records above and the client-row follow-ups in #741.)
C2 runner foundation: LANDED. The standalone delivery-aware runner lives
at tests/capacity_runner/ (issue #648, first runner PR). It accepts the full
contract input set (endpoint, seed, room/player count, protocol/encoding,
payload bytes, sender rate, delivery class, warm-up, duration, churn/reconnect
schedule, output directory; CAPACITY_RUNNER_* environment variables for
standalone capacity-host use), spawns the release server binary as a separate
process or connects to an external endpoint, schedules offered traffic
independently of response completion with a generator-lag bound, measures
scheduled send to same-clock recipient receipt on one monotonic run epoch,
and writes versioned artifacts: manifest.json (schema, run ID, config,
binary/config overlay hashes, toolchain, host, features, clock method),
deliveries.jsonl (tagged sends/receipts/gap reports/disconnects/faults
since schema 2),
intervals.jsonl (scraped delivery counters, server RSS, server and
generator CPU time, cgroup memory,
generator RSS, with unavailable counters recorded as null or explicit
scrape errors), summary.json (the oracle outcome), and
latency-histogram-v2.hdr. A replay
of the raw events reproduces the outcome summary exactly (artifacts::replay).
The small reliable relay scenario passes and every registered negative
control invalidates with its explicit reason: missing, duplicate, misrouted,
and out-of-order deliveries (deterministic oracle controls), a paused
generator (lag lands in scheduled-send latency, not reduced offered load),
generator saturation (explicit reason with worst lag and unsent work), server
termination (declared fault, gap-free prefixes preserved), and a slow reader
(eviction recorded and accounted by the server's
websocket_slow_consumer_disconnects_total counter).
C2 next runner PR: latest/volatile delivery classes — LANDED (second
runner PR, 2026-10-05). DeliveryClass carries the full lossy-class
contract: latest (keyed newest-value; latest_keys_per_sender selects
newest-value or key isolation) and volatile. Every omission must arrive
as an exact server-stamped gap report (latest_superseded,
latest_dropped_full, or volatile_dropped — a reason the class cannot
produce is a violation), gaps may not overlap each other or a delivery,
ranges may not reach beyond the sender's relay stream or below the first
1-based sequence, and reliable runs reject any gap at all. The oracle
validates coverage per recipient (uncovered holes are MissingDeliveries;
only the loud disconnect tail may stay uncovered) and the summary now
carries per-recipient latency tails plus gap_covered totals, and records
the run class's seven accountable server outcomes in every interval sample.
The pressure control runs both classes over real sockets. Each cell offers
16-KiB application payloads at 200 sends/s with a three-second read pause,
a 3.5-second measurement, a clamped receive buffer, and a tiny server send
queue. The pause still spans about 600 offered frames per sender. The lower
instantaneous rate reduces generator load while the longer pause retains
the declared byte pressure. The generator lag bound stays at 500 ms.
This byte volume replaces the old 96-byte control's fixed frame-absorption
estimate (#783).
Each run must stay valid, cover at least 100 omissions with exact gaps, and
match the server's per-class counter. The artifacts must replay exactly.
These large payloads are declared fault-control inputs. The C3 relay cells
still use 96-byte and 1-KiB payloads. Key isolation is pinned: distinct keys
coalesce nothing. Artifacts bumped to schema 2 (gap-report event kind,
class-aware summary). The e2e cells exposed one runner defect during the
red run: a peer's join snapshot can miss the member that joined
concurrently, so receivers now track PlayerJoined/PlayerLeft to resolve
gap senders.
Live runner cells share a test lock under plain cargo test. The coverage
and MSRV suites can otherwise run many generators at once and exhaust a
cell's lag bound. The lock matches nextest's process-spawning isolation;
the workload and validation limits stay the same.
Each connection polls reads alongside one persistent scheduled write. Ready reads and writes take turns, so overdue sends cannot starve receipts. Sender pauses hold only the write future. Reader resumption, churn, and quiescence wake independently of a pending write. Churn drops the old write and both socket halves before rejoining. The runner cancels and awaits its owned tasks before artifact capture. One immutable interval snapshot supplies both the published samples and the returned final counters (#783).
Windows CI exposed a failed-write termination race (#799) in session 362.
After the declared server kill, a socket write could fail before read EOF.
The runner stopped without disconnect evidence, so the oracle classified
one cutoff-tail delivery as missing. Transport write failures now stop
outbound work and drain buffered receipts with the existing read hooks and
quiescence deadline. Only EOF, close, or read error records a disconnect.
A pending drain produces a deadline fault. Ordinary write failures keep
SendFailed; the declared kill keeps ServerTerminated. Generator stops
retain their existing outcome. A deterministic failed-write control was
red when queued receipts were discarded. The fixed control covers terminal
and deadline paths, exact disconnected prefixes, interior holes, and
serialized evidence. The live termination cell checks artifact replay.
No oracle rule, artifact schema, or workload limit changed.
The runner builds its metrics client and prepares every peer before arming. Each peer registers its initial identity and reports readiness. Only then does the runner publish one shared measurement epoch. Setup consumes no send-lag budget. Scheduling delays after arming still count against the declared bound. Deterministic controls cover delayed setup and delayed traffic. Invalid pressure runs print recent send and sample timestamps.
PR #789
landed the readiness gate. Fresh main CI
on af9616a8 passed Linux, macOS, Windows, MSRV, and coverage verification.
C3 capacity measurements remain pending.
C2 next runner PR: churn/reconnect schedules — LANDED (third runner
PR, 2026-10-05). ChurnSchedule carries the reconnect-burst storm (the C3
reconnect cell): at start, the seed-chosen fraction_percent of peers
disconnect; each rejoins at a seed-staggered instant inside [start,
start + window), and every victim's post-disconnect sends shift by its
offline window (a scheduled gap is workload shape, never generator lag).
The stream identity across a storm is the runner's own incarnation index —
one join per connection, registered per PlayerId in a shared sender
registry that receiving tasks resolve every inbound frame and gap report
through — because the server's relay stamps are per connection and a fresh
rejoin is a new member whose (epoch, seq) restarts at (1, 1). The
oracle validates per (recipient, sender, incarnation) streams: rejoin
snapshots' per-member (id, seq tail) stamps raise each stream's owed
floor (the loud away window), stale-epoch deliveries after a sender's
rejoin and below-tail deliveries after a recipient's rejoin are misroutes,
and a planned storm that never fires invalidates the run. Artifacts bumped
to schema 3 (receipts carry the sending incarnation and per-connection
server sequence, sends carry their incarnation, churn events record the
disconnect/rejoin cycle with the rejoin snapshot tails). One real-socket
storm cell (50% of a four-peer room, reliable) passes with replay
equality; deterministic controls pin each new permitted outcome, and the
red proof neuters the stale-epoch check (the control fails). A schedule
control pins the offline-window shift exactly (moved sends land at or
after their reconnect instant with count and spacing preserved,
non-victims untouched), and adversarial-review hardening keeps snapshot
floors authoritative over later rejoin events, requires the rejoin half
of every planned cycle, keeps the stagger window below the generator-lag
bound, and refuses run-level fault hooks in churn cells. The bot review
round hardened identity further: rejoin snapshot tails are recorded
UNRESOLVED and resolved against the registry recorded at end of run (the
per-frame registry lookups are race-free because a peer registers its id
before its first send; the per-tail lookups were not, and a wrong guess
floored the wrong incarnation), a member omitted from a rejoin snapshot
(the documented join-snapshot race under concurrent rejoins) is neither
closed nor unfloored — its away window is derived from the sends that
completed at or before the rejoin instant, a rule that errs safe —
derived floors are bookkeeping only (only snapshot tails are
server-enforced watermarks, so only they turn a redelivery into a
misroute), closed streams are finished (nothing further owed, arrivals
still misroute), and the default burst window (200 ms) sits strictly
below the default generator-lag bound (250 ms) so the default env config
runs. Churn runs require the v3 wire and exclude the socket-owning and
generator-latency hooks.
C2 next runner PR: room-replacement churn — LANDED (fourth runner
PR, 2026-10-05). ChurnSchedule carries the room-replacement shape (the
C3 churn cell): on every wave start + k * interval — while the wave fits
the scheduled-send span — a seed-chosen fraction_percent of whole ROOMS
cycles: every member disconnects at the wave instant and rejoins, seeded-
staggered inside the window, into the room's NEXT generation. The
generation is a fresh six-character alphanumeric room code
(RunConfig::room_code_for_generation: generation 0 keeps the decimal
{prefix}{room:03} code; generation g ≥ 1 is
{prefix}{letter}{room in two base-36 digits} — the leading lowercase
letter is a character class the decimal generation-0 suffixes never start
with, so the classes are structurally disjoint, (letter, room) is
injective for rooms below 36², and the alphabet caps a room at 26
generations), so the replacement creates a genuinely new room and the
now-empty old room tears down — room creation/destruction churn, not a
rejoin. Because the whole room moves together, the member roster per room
is unchanged, so the oracle's static co-room checks and
per-(recipient, sender, incarnation) stream machinery validate
replacement runs unchanged; the member set a seat expects never changes,
only the code. Replaced-room stream identity, fresh-room rejoin snapshots
(the first seat's snapshot is empty — the room starts fresh; a later
seat's snapshot tails the members that rejoined before it), and the
derived-floor rule for members a snapshot omits all ride the existing
controls. The multi-wave shift composes correctly: a send moves by the
offline duration of exactly the waves whose disconnect it was originally
due past (the comparison runs on the original timeline, so a send due
between two waves does not inherit the later wave's shift), pinned per
send. Coherence refusals: the window stays below the interval (waves
never overlap, so a room is whole again before the next selection), the
first wave and every wave must fit the scheduled-send span, a room's
deterministic victimization count stays within its code space (the cap
rides the built plan's per-room count, not the wave count, so the C3
wide-low-fraction shape is never refused for a bound no room reaches),
and clock overflow is a refusal, not a panic; the
lag-bound refusal now covers every churn shape's window, and the
rejoin-half requirement counts one rejoin per planned victimization (a
two-wave plan with one rejoin per member is named invalid per member).
Two real-socket cells pass with replay equality: a two-room run where
half the rooms are replaced per wave (whole rooms cycle to new
incarnations while the other rooms keep serving, no cross-room leakage)
and the burst storm cell; deterministic controls pin the replacement
plan shape (whole-room victim sets, wave instants, in-window stagger,
determinism), the composed shift, generation-code uniqueness over the
full enforced room range plus the structural class disjointness, the C3
shape admission, and the three outcome families (valid replacement,
missing rejoin half, stale-generation misroute). Red proofs: the
missing-rejoin control fails when the per-victimization count is
weakened to any-rejoin, and the plan control plus the e2e cell fail when
the replacement plan builder is neutered to an empty plan; the
stale-generation control rides the stale-epoch check the third PR
red-proved. An adversarial review round caught the first encoding draft
(g * 1000 + room in three base-36 digits) aliasing live generation-0
codes — base-36 values at and past 36² can be all-decimal, so
(room 296, gen 1) collided with room 100's initial code; the
letter-first encoding replaced it, and the uniqueness control now sweeps
every room 0..999 × generation 0..=26 instead of sampled rooms.
C2 next runner PR: richer resource counters — LANDED (fifth runner PR,
2026-10-05). Every interval sample now carries the resource counters the C3
capacity claims are read from, beside the delivery counters: the
ingress/egress byte pair and the queue-posture gauges.
signal_fish_websocket_egress_bytes_total counts the application payload
length of every successfully written frame — charged at the write leaf after
the sink accepted the frame, so an egress byte is a byte that reached the
connection; ping/pong/close carry no application payload and are not counted,
mirroring the sender-side signal_fish_relay_bytes_total admission counter.
signal_fish_websocket_queue_depth (items resident across every live
classified outbound queue) and
signal_fish_websocket_queue_oldest_age_milliseconds (age of the oldest
resident item against the scrape instant) are walked over live connections
at scrape time — the runner samples them exactly when the endpoint is hit,
and nothing on the write path maintains them. Unavailable counters stay
recorded-as-null; the spawned binary exposes all four, so the acceptance
scenario now pins per-sample presence, byte-counter progress, and fan-out
amplification (egress strictly above ingress for the four-peer room). Red
proofs: neutering the write-path increment failed the exact-egress socket
test (counter must equal the payload bytes the client observed); neutering
the queue walk failed the connection-manager sample test (depth, oldest-item
stamp, and scrape-instant age); renaming the rendered series failed the
gauge render test.
C2 next runner PR: server and generator CPU-time accounting — LANDED
(sixth runner PR, 2026-10-06). Every interval sample now carries
server_cpu_seconds and generator_cpu_seconds beside the RSS pair:
cumulative utime + stime from /proc/<pid>/stat at the kernel's fixed
USER_HZ = 100 stub, read off-path at scrape time for the spawned server
PID and the runner process itself. This is the C2 resource-collection
remainder's CPU pair — the C3 comparison of generator cost against server
saturation needs both processes' consumed CPU, and the runner's own cost
must be distinguishable from the server's. Unavailable values stay recorded
as null (off-Linux hosts). Artifacts bumped to schema 4 (additive interval
fields; a manifest naming schema 3 or older is refused by the validator).
Red proof: neutering the sampler wiring failed the acceptance scenario's
per-sample presence pin; the fixture parser test pins the spaced-comm field
positions, and the live-sampler test pins strict advancement under real CPU
work with absent PIDs recorded as null.
C2 next runner PR: unsupported-format contract-experiment cells — LANDED
(seventh runner PR, 2026-10-06). The runner gains a labeled Experiment
input (unsupported-format): peer 0 of every room negotiates opaque rkyv
(overlay knob protocol.enable_rkyv_game_data, refused at config time if
absent and re-verified against the server's advertised ProtocolInfo list
so a silent downgrade cannot hollow the cell) and sends raw binary frames;
every other peer stays JSON. The oracle validates the cross-format refusal
family per the delivery contract instead of reliable semantics: the opaque
stream must reach NO recipient as a payload (any arrival, or any binary
frame anywhere, is the unsupported_format_leak class), every omission must
be covered by exactly one exact unsupported_format gap report (a hole
without its report is missing_deliveries), a foreign reason on the opaque
stream is invalid, the gap contract stays closed on the text streams, and
the rate-limited advisories are recorded evidence bounded at one per opaque
sender per second (unsupported_notice_flood names the exact count and
bound). Inbound classification is exact: advisories at cross-format
observers are notices, the same advisory at the opaque sender (and any other
error code) stays a rejection. The run is refused outside v3/reliable/
churn-free shapes; binary frames carry no delivery class. Summary and
manifest carry the experiment label, notices are summed in the summary, and
artifacts bump to schema 5 (new event kind unsupported_notice, additive
summary fields; older schemas refused by the validator). The real-socket
cell (1 room, 1 opaque + 3 JSON observers) pins contract legality, exact
coverage (gap_covered == observers × sends), zero opaque receipts, the
labeled replay, and exact server-counter agreement
(class_outcome_unsupported_format == observers × sends, now sampled for
experiment runs too). Six deterministic controls carry red proofs: the
leak rule, the opaque-stream coverage rule, the foreign-reason scope, the
text-stream scope, the notice-cadence bound, and the inbound
classification each fail their control when neutered. Remaining
unsupported-format ground (tracked for later cells, not silently skipped):
v2 observer cohorts (advisory-only, no reports to validate), same-format
opaque twins (binary-envelope decoding), and churn × experiment composition.
C2 next runner PR: live-state, backlog, maintenance, and socket-memory resources — LANDED (eighth runner PR, 2026-10-07). The resource-collection remainder is closed; every interval sample now carries the four missing resource families beside the delivery counters and the RSS/CPU/cgroup set:
- Live objects (scrape-time, never write-path):
signal_fish_rooms_live,signal_fish_room_occupants_live,signal_fish_reconnection_pending,signal_fish_replay_rooms_retained, andsignal_fish_replay_events_retained, read under short read locks when the endpoint is hit (GameDatabase::live_room_counts, defaultNonefor embedder backends that cannot answer cheaply, andReconnectionManager::replay_sample). An unavailable count renders no series — absence stays distinguishable from a fabricated zero. - Cleanup backlog:
signal_fish_cleanup_pending_publications, the rooms in the pending-publication lifecycle at scrape time (in-flight creation or awaiting repair; maintenance drains the abandoned ones). - Maintenance cost:
signal_fish_maintenance_sweeps_total(completed sweeps) andsignal_fish_maintenance_last_duration_milliseconds(the most recent completed sweep's monotonic wall duration) — the C3 maintenance-complexity signal. Drain-aborted passes are not counted. - Socket memory:
server_socket_tcp_mem_pagesandserver_socket_udp_mem_pagesin every interval sample — TCP+TCP6 and UDP+UDP6memfrom/proc/<pid>/net/sockstat, in kernel memory pages (the page size is a capacity-host environment fact),nulloff-Linux. AFRAG:line'smemoryfield is never swept into the totals.
Artifacts bumped to schema 6 (additive interval fields). Red proof:
neutering the sweep recording call fails
maintenance_sweep_accounting_records_every_completed_sweep (the counter
never advances). Presence pins: the spawned binary's run asserts all eight
new series as u64 in every interval sample on every platform, and the
socket-memory pair as recorded on Linux (null elsewhere, mirroring the
CPU-pair contract); renaming any rendered series fails the render pins.
C3 measurement prerequisite review — 2026-10-07¶
In the runner shipped by PR #768, the latency oracle subtracted completed socket-send time from receipt time. This excluded generator lag and socket-send wait from the scheduled-send SLO. Both histogram paths also clipped samples above 60 seconds. #774 fixes this measurement defect. No server runtime defect is claimed.
Schema 7 uses intended send time for aggregate, per-recipient, and histogram latency. Histogram bounds cover the observed range; the summary maximum is exact. Replay rejects earlier schemas so historical results cannot silently acquire corrected semantics. The deterministic control covers all three delivery classes, warm-up exclusion, delayed sends, receipt before completed send, and 70-second stalls. The real-socket pause control requires the delay in each recipient's latency and verifies replay equality. The deterministic test failed on the old calculation before the fix.
Large C3 runs remain gated on #775 (generator memory) and #776 (effective configuration and payload-size provenance). Required encoding and idle-lobby cohorts, constrained-host setup, and external-host resource collection also remain. No capacity point or deployment claim is accepted by this review.
C3 generator latency memory experiment — 2026-10-07¶
The first part of #775 removes per-delivery recipient clones and latency vectors. The oracle now feeds aggregate and recipient HDR buckets in one pass. The artifact writer also streams samples. Recipient buckets are allocated only when measured receipts arrive. Schema 7, exact maximum latency, delivery verdicts, and replay stay the same.
The registered experiment compared main after
PR #777
with this change. Both debug test binaries used the same synthetic complete
reliable fan-out: 16 peers, with 100, 1,000, or 5,000 sends per peer. Each cell
ran three times in a fresh process, baseline first, then candidate. Linux
os.wait4 supplied peak child RSS in KiB. The initial GNU time method was
replaced before measurements because that tool was absent. All attempts were
retained. Full summary JSON matched the baseline for every candidate run.
| Receipts | Baseline median KiB (range) | Candidate median KiB (range) | Reduction |
|---|---|---|---|
| 24,000 | 15,364 (15,360–16,676) | 14,052 (14,048–14,052) | 8.5% |
| 240,000 | 68,692 (68,560–68,764) | 52,276 (52,020–52,336) | 23.9% |
| 1,200,000 | 312,756 (312,756–312,824) | 223,852 (223,832–223,960) | 28.4% |
The host used ARM64 Linux under WSL2 and rustc 1.91.0. These numbers measure synthetic generator memory in the debug profile. They establish no server capacity point. The generator had no CPU or memory constraint.
To repeat, build cargo test --test capacity_runner --no-run and run its
executable in a fresh process under a peak-RSS profiler:
CAPACITY_MEMORY_PROBE_SENDS=5000 <test-executable> \
generator_memory_profile_on_complete_fanout --exact --ignored --nocapture
Use send counts 100, 1000, and 5000. The ignored probe prints the complete
summary, collection RSS, and oracle time. Registrations, raw outputs, resource
records, and comparisons are retained locally in
/tmp/signal-fish-c3-generator-memory-20261007/.
Raw event storage, sender schedules, and delivery-validation indexes still grow with the workload. Issue #775 still requires those structures to stream or stay bounded before the required long C3 cells can run.
C3 spawned-server configuration provenance — 2026-10-07¶
The configuration part of #776 now records the full typed settings for each spawned server. Schema 8 retains compiled defaults, the harness base, the declared overlay, the final effective configuration, and its SHA-256 over compact JSON. The manifest also hashes the actual launched binary. The overlay hash remains a hash of declared input.
The harness launches a hard-linked binary, or a copy when linking fails, in its
isolated temporary directory. This prevents a config.json beside the build
output from contributing unrecorded map entries. The child environment removes
inherited SIGNAL_FISH* variables. The reserved port overrides any overlay
port. Legacy app-access keys retain the production loader's meaning.
Capacity runs reject file-backed app registries and connect-token public keys before starting. Those sources change configuration after JSON merging; their contents are not yet recorded. Use inline values for these runs. Replay checks config hashes, layer reconstruction, port agreement, and server identity. These checks detect inconsistent artifacts; they are not signatures.
External endpoints without supplied evidence are labeled unknown_external.
Their delivery results can remain valid and replay exactly, but they cannot
support an accepted capacity point. Schema 10 adds the
external host evidence intake. Schema 9's exact
application and encoded frame-body size evidence is described below.
The real-binary regression compares the snapshot with the production
--print-config loader under the same isolated configuration and environment.
It pins defaults, harness overrides, legacy keys, and the reserved port.
Additional controls reject malformed overlays, conflicting aliases, file-backed
sources, altered hashes or layers, and old schema 7 artifacts. The manifest
presence check failed against the previous runner before acceptance.
External host evidence¶
Issue #776
requires external-host config and binary identity before capacity acceptance.
Schema 10 embeds an operator capture as external_host_declared. Runs without
this capture retain unknown_external. The delivery oracle's verdict remains
separate from provenance and from the capacity acceptance gates.
On the server host, use the deployment's user, working directory, environment,
and exact binary. Freeze config, referenced files, and the binary for capture
and the entire run. For stdin-based deployments, preserve the same inputs in a
config file first. The
capture sample
validates settings, invokes --print-config-evidence, hashes the binary, and
hashes referenced app-registry, connect-token key, and active TLS files.
It refuses to overwrite an earlier capture.
python3 .llm/code-samples/capacity/capture-host-evidence.py \
--binary /srv/signal-fish-server \
--endpoint ws://capacity-host:3536 \
--deployment immutable-deployment-1 \
--output /tmp/host-evidence.json
Copy that file to the generator. Set CAPACITY_RUNNER_ENDPOINT to the exact
endpoint in the capture and CAPACITY_RUNNER_EXTERNAL_HOST_EVIDENCE to the
file path. The endpoint must be a ws or wss origin with an explicit port
and no path, credentials, query, or fragment. Proxy listener ports may differ
from the backend config's port.
The runner checks complete canonical redacted settings, the snapshot hash, SHA-256 formats, a positive binary size, host and deployment identifiers, the collection timestamp, and exact referenced-file hash coverage. It embeds the document in the run inputs and server provenance. The original host file is not needed for replay. Invalid evidence fails before output or connections. Replay also checks endpoint, binary identity, and input/provenance agreement. Earlier schemas are rejected.
The snapshot hash covers the redacted config. The loaded-config hash covers
all loaded values, including secrets and file-loaded app and key values.
File hashes identify referenced authentication and active TLS material without
archiving its contents. Keep captures and fingerprints private. Neither hashes
nor a second loader invocation authenticate the live remote process. The
operator must verify that the captured deployment serves the endpoint and
does not reload or change during the run. Capture again for a changed deployment.
has_config_evidence reports evidence presence, not accepted capacity.
Controls cover a real host binary and collector, source-file removal before the run, exact delivery replay, unknown-host replay, endpoint and identifier boundaries, secret redaction and rotation fingerprints, file-hash coverage, malformed inputs, inconsistent archived identities, and schema rejection. This intake establishes no capacity point. The generator-memory and macOS delay requirements in #775 and #795 remain open.
C3 exact application and encoded frame-body sizes — 2026-10-07¶
The payload part of
#776
now uses exact application byte counts. payload_bytes includes the compact
JSON bytes of the complete ledger document: sender, sequence, JSON keys, and
padding. Padding shrinks as sequence numbers gain digits. A run is refused
before artifacts, connections, or child processes when its target cannot hold
the largest scheduled metadata document.
Schema 9 records application bytes and actual encoded WebSocket message-body bytes on every completed send and observed ledger receipt. Encoded bytes include protocol envelopes, class/key fields, sender IDs, and server stamps. They exclude WebSocket headers and masking, TCP, and TLS. The opaque experiment sends the ledger document directly, so its ingress application and body sizes are equal.
The summary separates warm-up and measured ingress/egress. Unknown keys and unidentified GameData remain in unmatched egress. Each population records count, total bytes, and minimum/maximum sizes. Duplicate arrivals still contribute bytes. Missing or forged ledger identities produce structured faults with their observed sizes. Every observed application size must match the configured target; invalid body sizes and total overflow invalidate the run. Overflow totals saturate with an explicit fault so JSON serialization remains exact.
Receipt time is captured when the runner receives the frame, before decoding and byte measurement. Scheduled-send latency still includes generator and socket wait. Schema 9 identifies this receipt-boundary change and the byte evidence; replay rejects earlier schemas and reconstructs the same summary from raw events. Hashes and counters do not replace the recorded per-frame sizes.
The baseline real-socket control observed 141 application bytes in a declared 96-byte cell and failed the exact-size assertion. The corrected eight-cell control covers 96 and 1,024 bytes for v2 reliable JSON and v3 reliable/latest/ volatile JSON. It verifies exact application sizes, encoded ingress/egress lengths including metadata, phase totals, and replay equality. Deterministic controls cover decimal sequence boundaries, escaped sender names, maximum ledger sequence, an initial sequence that fits but later metadata that does not, wrong warm-up sizes, duplicate and unknown arrivals, unidentified frames, and overflow. Pressure and unsupported-format controls retain their delivery contracts.
External-host configuration and binary evidence intake remains in #776. Generator memory, encoding cohorts, resource constraints, and the long capacity cells remain separate prerequisites. No server capacity point is claimed here.
C3 generator schedule memory — registered experiment¶
Issue #775 remains open. This experiment isolates schedule construction. It
compares main 735734ca with compact schedules using the same ignored
generator_schedule_memory_profile probe. Each size runs in three fresh Linux
processes. Python os.wait4 records peak child RSS in KiB. Record every attempt,
exit status, raw output, and wall time. Do not retry failed trials silently.
Use seed 1, two rooms, 16 players per room, 60 sends/s, 120 seconds of warm-up, and measured durations of 60, 600, and 3,600 seconds. The workload uses the runner's integer-microsecond period. Compare full schedule counts, a checksum of every scheduled event, and exact first, phase-boundary, and final events for every sender. Require identical output between baseline and candidate.
Then build the candidate's 999-room, 16-player shape with 120 seconds of warm-up and 600 seconds of measurement. Sample the same event positions and record its count and RSS. This constructs a ten-minute schedule; it does not send traffic or prove a ten-minute live capacity cell. Event retention, oracle indexes, replay inputs, and churn plans require separate memory controls.
The experiment ran on ARM64 Linux under WSL2 with rustc 1.91.0 in the debug test profile. All 21 processes exited successfully. Each paired size produced identical probe output in all six baseline/candidate runs.
| Scheduled sends | Baseline median KiB (range) | Compact median KiB (range) | Reduction |
|---|---|---|---|
| 345,632 | 16,608 (16,352–16,608) | 8,904 (8,848–8,904) | 46.4% |
| 1,382,464 | 40,916 (40,912–40,928) | 9,272 (9,068–9,480) | 77.3% |
| 7,142,688 | 175,840 (175,840–175,840) | 9,868 (9,676–10,148) | 94.4% |
The larger candidate shape contains 690,540,768 scheduled sends. Three fresh
processes had median peak RSS of 201,916 KiB (range 201,872–201,924). The probe
retains and serializes four sampled events per sender. Peak RSS includes this
artifact buffer and process overhead; it is not a count of schedule bytes.
The integer-microsecond period produces 43,202 sends per sender for this shape.
Raw outputs, resource records, and binary hashes remain under
/tmp/signal-fish-c3-schedule-memory-20261007/.
Each sender now stores scalar schedule inputs and its churn windows. Indexed SplitMix access preserves the original timestamp and phase at every sequence. Churn shifts compare against the original timestamp. Payload-size validation and first-measured-send lookup use direct indexed access. Invalid clock and peer inputs fail before plans are built. Impossible replacement-wave shapes fail before wave allocation.
An independent frozen eager reference covers seeded jitter, warm-up boundaries, partial periods, indexed access, and churn. It rejected an intentional PRNG index mutation. Fifteen focused controls passed, including real-socket pressure, reconnect, replacement, exact payload sizes, and artifact replay. Retained events, oracle indexes, replay memory, and the ten-minute live cell remain open in #775.
C3 replay input memory — registered experiment¶
Issue #775 remains open. This experiment isolates JSONL input buffering. Compare
main 5414c7fe with streaming readers using the same ignored
generator_memory_profile_on_complete_fanout probe. Generate each input artifact
once, outside the measured reader processes, from 16 peers with 100, 1,000, and
5,000 sends each. The files contain 24,000, 240,000, and 1,200,000 receipts plus
the corresponding sent events. Record file sizes and SHA-256 hashes.
For each input, run three fresh baseline readers, then three candidate readers.
Use Python os.wait4 to record peak child RSS in KiB. Each reader rebuilds the
same compact plans, reads the same artifact, and runs the exact oracle. Require
full summary JSON equality across all paired runs. Retain every attempt, exit
status, raw output, and wall time. Do not retry failed trials silently.
The input is synthetic reliable fan-out. No server traffic or capacity point is measured. Streaming removes the whole-file byte buffer; retained records, oracle indexes, and the largest JSONL line remain separate memory costs.
The experiment ran on ARM64 Linux under WSL2 with rustc 1.91.0 in the debug test profile. All three fixture producers and all 18 measured readers exited successfully. Every reader reproduced the producer's full summary JSON.
| Receipts | Input bytes | Baseline median KiB (range) | Streaming median KiB (range) | Reduction |
|---|---|---|---|---|
| 24,000 | 4,270,383 | 16,712 (16,572–16,712) | 15,864 (15,736–15,864) | 5.1% |
| 240,000 | 43,421,095 | 90,056 (89,928–90,312) | 57,400 (57,344–57,408) | 36.3% |
| 1,200,000 | 220,485,095 | 419,912 (419,896–420,040) | 243,568 (243,552–243,956) | 42.0% |
Input hashes, binary hashes, producer records, raw reader output, and every
measured attempt remain under /tmp/signal-fish-c3-replay-memory-20261007/.
Both JSONL readers now reuse one line buffer. The parsing rules, event order,
arrival order, and schema remain unchanged. The buffer retains the capacity of
the largest line; parsed records and oracle indexes still grow with workload.
Controls cover all event kinds, empty lines, CRLF, a final line without a newline, repeated registry entries, duplicate JSON keys, malformed events, large Unicode lines, and late read errors. A mutation that reads the whole file failed the malformed-prefix control because it read the protected tail. Real-socket payload, churn, and replay controls also passed. No live ten-minute cell or accepted server capacity point is proved by this experiment.
The same I/O review found #793:
both JSONL writers could hide a final buffered write failure. A Linux /dev/full
control showed both writers returning success before the fix. Both now flush
explicitly and report errors with the artifact path. The repository sweep found
two other Rust buffered writers, both already flushing. The failure control and
real-socket artifact/interval replay control passed after the fix.
C2 generator delay investigation — #795¶
The scheduled main run
37619144139
failed the macOS payload-cell test on commit 5414c7fe. The
V3Json / Volatile / 1024 cell recorded 417,628 microseconds of generator lag
against a 250,000-microsecond limit. The test rejected the run. Its temporary
artifacts did not survive the test failure.
The full PR run
37621168700
and the main run for commit 38b76ee8
37623685271
passed the same test on macOS. These passes do not explain or fix the earlier
delay. #795
remains open.
Two independent source reviews found no confirmed cause. The task readiness gate precedes the shared epoch. Read and write polling alternates, and lifecycle deadlines take precedence. The nextest process group already isolates capacity tests. On macOS, in-run CPU and RSS probes return unavailable values; manifest subprocesses and binary hashing finish before the epoch.
The old failure message cannot distinguish the runner's rejection before a
write from the oracle's rejection after a completed send. Both report
GeneratorSaturated. The runner now emits structured stderr evidence at both
paths. It records sender, sequence, phase, intended time, observed time, and
bound. Completed writes also record preparation and write elapsed times. Write
elapsed time includes task scheduling; it does not prove socket backpressure.
Before artifact I/O, an invalid result emits the run identity, workload, runtime worker count, declared fault hooks, fault and reason counts, and bounded context. It keeps the first, last, and worst completed send for every configured peer, including peers with no completed sends. It keeps the largest sampler gap and three recent samples. Unavailable CPU and RSS values stay null. Fault and reason previews contain at most 16 entries each, with total counts to make truncation explicit. Endpoint, server config overlay, and full counter maps are excluded. Free-form error details are omitted from diagnostic previews. Join errors keep their count, and samples report scrape failure as a flag. The artifacts retain full errors.
The lag limit, offered schedule, invalid-run checks, payload assertions, replay results, and schema remain unchanged. This evidence supports investigation; it does not establish a cause.
Six data-driven controls cover both timing paths, every configured peer, empty peers, unavailable resources, sampler gaps, preview bounds, and declared hooks. A mutation that read only the final eight sent records failed the all-peer control. The old binary passed the stall and replay control but emitted no timing evidence; the new binary captured all three stalled peers.
A temporary 600-ms delay before the final socket write captured four completed
write observations. No raw runner fault was recorded; the oracle rejected the
completed-send lag against the unchanged 250-ms limit. The source was restored
byte for byte. All eight healthy payload and replay cells passed and emitted no
failure diagnostics. These controls prove capture and validation, not the
cause of the earlier macOS delay. Raw proof files remain under
/tmp/session361-*.log and /tmp/session361-*-proof.json.
C2 inbound evidence after generator stop — #814¶
The runner returned from the peer task when scheduled-send lag or frame preparation stopped outbound work. This dropped both socket halves before quiescence. The oracle still treated that recipient as connected because no termination was observed. This could truncate receipt evidence; it does not establish a server delivery defect.
The peer now stops outbound work for its entire lifetime. It keeps the socket, normal inbound handling, read pauses, and declared churn until shared quiescence or observed termination. A rejoin keeps the stop state. Transport write failures retain their existing drain and termination deadline.
The sweep covers lag refusal, application-data preparation, key conversion, and frame encoding. Completed writes still record their send and size data. The offered schedule, unsent accounting, generator invalidation, receipt clock, artifact schema, and oracle rules remain unchanged.
A real-socket peer-task control fails before the fix at the session-lifetime assertion. Six cases cover preparation failure and saturation with normal quiescence, server close, and rejoin. They retain held and later receipts, exact gap reports, and server rejections. They check original-clock receipt times, stopped sends after rejoin, unchanged saturation lag, and event-artifact round trips. Existing saturation and termination cells check full replay.
Test progress reads event counts without taking the evidence. The former
snapshot method now has the name take_records, which makes its ownership
transfer explicit. The final capture still transfers each record once after
all tasks finish.