Skip to content

Scaling Architecture

Signal Fish Server scales up within one process. The shipped server does not provide transparent horizontal scaling: all room and reconnect state is in-memory, and a WebSocket is assigned to a process before its JoinRoom message supplies a room code. Read the single-instance deployment contract before placing the server behind a load balancer.

For the exact operation acknowledgement, process-loss, and reconnect gameplay loss boundaries, see the consistency and durability contract.

What state a process holds

A process keeps all authoritative state in memory:

  • rooms, room codes, players, spectators, lobby and ready state;
  • one registered delivery route per connected client;
  • reconnection records, replay events, and token claims;
  • relay (epoch, seq) counters and delivery-accountability state; and
  • v3 session plans and transport status.

Every forwarding decision reads that local state. Relay-floor GameData fans out over local channels; WebRTC Signal validates both peers against the local room; session plans are computed from local membership. No shipped component replicates those facts to another process.

The room is the conceptual scaling unit

Rooms are independent within one process: two rooms never exchange messages, and v3 enforces same-room signaling on every hop. That makes a room the natural unit for a future external routing layer, but Signal Fish does not implement that layer.

An application-owned directory may assign rooms to separate, isolated Signal Fish deployments and give clients a deployment-specific WebSocket URL before they connect. server.room_code_prefix can make generated codes carry a deployment hint (for example EU7K2X), but the prefix is only a hint: the server does not resolve it, redirect a WebSocket, or prevent another process from creating the same code. Adding or removing homes while rooms are live requires application-level drain and rebuild, not consistent-hash remapping.

Directory-driven joins must be collision-safe. A routed join sets join_only: true on JoinRoom (issue #625): the join then refuses ROOM_NOT_FOUND when its code does not resolve on that home, so a stale directory entry surfaces as a refusal the client can re-resolve instead of a second room silently opening under the same public code. The directory, not the server, owns creation on the chosen home.

Two consequences follow:

  • Relay and signaling share the same boundary. A room's GameData, WebRTC signals, reconnects, and re-plans must all use its one home process.
  • WebRTC can reduce server bandwidth after P2P establishment, but it does not make room control state portable. The WebSocket relay remains the fallback.

Cross-process fan-out is not implemented

A message bus and shared database would be necessary but not sufficient for a room to span processes. The code contains extension seams:

  • MessageCoordinator abstracts send-to-player and broadcast-to-room;
  • SequencedMessage is an envelope with origin and targeting metadata; and
  • DistributedLock abstracts room-operation locking.

Their shipped implementations are in-memory. They do not provide a shared room directory, consensus, sequence ownership, global deduplication, reconnect token handoff, or cross-process delivery. Treat them as local coordination tools and future seams, not as distributed behavior.

Multi-region hints

server.region_id and server.room_code_prefix anticipate an external routing layer:

  • every room/player record carries the local region internally; and
  • generated room codes may carry a deployment-specific prefix.

Neither changes protocol behavior. A prefix or region ID does not make a reconnect token routable and does not establish room affinity by itself.

Capacity drivers

  • Connections and rooms: see the resource requirements for current per-process guidance.
  • Relay-floor bandwidth: rooms that do not upgrade to P2P keep all GameData on the process, so message rate, payload size, and recipient fanout dominate network and queue pressure.
  • P2P signaling: after WebRTC establishment, the process normally carries only control, signaling, and fallback traffic.
  • TURN: TURN relay bandwidth is operated separately from the signaling process; see the TURN deployment guide.

These are operational starting points, not universal saturation guarantees.

Size the relay floor

Relay cost grows with fan-out, not just sender rate. For a room with N players where every player sends S messages per second, each client receives S × (N - 1) messages per second and the server performs:

Text Only
recipient deliveries/second = N × S × (N - 1)
payload egress/second        = deliveries/second × payload bytes

WebSocket, protocol, JSON/MessagePack, IP, and TLS framing add to the payload floor. Measure the encoded frames for capacity planning; do not provision only the application payload number.

The H2 exact-ledger experiment fixes N = 16, a 1 KiB application payload, JSON relay, production queue defaults, and one second at each sender rate. It requires every (receiver, sender, sequence) delivery, valid v3 stamps, no slow-consumer eviction, and reports p50/p95/p99/max latency, backpressure, wall time, and process RSS. The registered load grid is:

Per-player send rate Per-client receive rate Server recipient deliveries Application-payload egress floor
30 msg/s 450 msg/s 7,200/s 7.03 MiB/s (56.3 Mbit/s)
60 msg/s 900 msg/s 14,400/s 14.06 MiB/s (112.5 Mbit/s)
120 msg/s 1,800 msg/s 28,800/s 28.13 MiB/s (225 Mbit/s)
240 msg/s 3,600 msg/s 57,600/s 56.25 MiB/s (450 Mbit/s)
480 msg/s 7,200 msg/s 115,200/s 112.50 MiB/s (900 Mbit/s)

The first exact-head GitHub run measured the following on one shared-process runner (server plus all 16 load clients):

Target per player Writer completion Observed deliveries p99 latency Queue backpressure RSS
30 msg/s 1.001 s 6,874/s 45 ms 0 22.8 MiB
60 msg/s 1.018 s 13,505/s 54 ms 0 23.1 MiB
120 msg/s 1.591 s 16,965/s 149 ms 0 24.2 MiB
240 msg/s 2.525 s 17,818/s 851 ms 0 26.1 MiB
480 msg/s 3.334 s 19,595/s 2,722 ms 0 28.2 MiB

Every one of the 223,200 sweep deliveries was exact and no client was evicted. The observed knee lies between the 120 and 240 msg/s targets: doubling offered fan-out from 28,800/s to 57,600/s increased completed throughput by only 5.0%, writer completion slipped beyond the scheduled second, and p99 rose from 149 ms to 851 ms without filling the server's outbound queues. At the 480 target p99 reached 2.72 seconds. On this shared-process experiment the load clients/socket ingress became the limiting boundary before the delivery queue. It does not establish a standalone server maximum.

The bounded PR-lane matrix already proves the 30 msg/s point across JSON and MessagePack with 2, 8, and 16 players: 17,880/17,880 deliveries, zero default- queue backpressure, zero eviction, and measured p99 of 20–55 ms. Treat the higher-rate nightly sweep as a trend measurement for its GitHub-hosted runner, not a portable capacity promise. Benchmark the release binary on the intended CPU, kernel, TLS termination, and network path, and retain headroom below the first point where p99 or queue pressure bends sharply.

Queue and freeze budget

For one recipient, estimate queue-fill time in messages, using encoded-message rates:

Text Only
drain_messages_per_second = drain_bytes_per_second / encoded_frame_bytes
excess_rate = max(0, incoming_messages_per_second - drain_messages_per_second)
queue_fill_seconds = send_queue_capacity / excess_rate
fail_loud_bound ≈ queue_fill_seconds + slow_consumer_timeout

queue_drain_seconds =
    (socket_bytes_ahead + send_queue_capacity * encoded_frame_bytes)
    / drain_bytes_per_second
available_queue_bytes =
    max(0, drain_bytes_per_second * max_sojourn_seconds - socket_bytes_ahead)
max_capacity_at_sojourn =
    floor(available_queue_bytes / encoded_frame_bytes)

If excess_rate is zero, the queue does not fill under steady state; still reserve burst headroom. A full reliable queue can pace room senders for at most one slow_consumer_timeout_ms window before that recipient is closed loudly. max_sojourn_ms independently bounds oldest reliable queued/batched work and each control item's own queue age through write completion. Latest/volatile queue age is handled by their loss policy; the value bounds their selected socket write instead. Any of those deadlines may close the connection first. socket_send_buffer_bytes bounds data already handed to TCP ahead of later control (the 65536-byte default is a request; operating systems may clamp or account it differently).

Do not multiply send_queue_capacity × slow_consumer_timeout and interpret the result as seconds: queue slots and time have different units. With the defaults, 1024 × 5000 ms is only a message-time exposure measure; the actionable time bound is queue fill plus the 5-second capacity wait. Raising queue capacity absorbs a larger burst but retains more stale reliable work. Raising the timeout tolerates a longer drain pause but directly raises the maximum room freeze after the queue is full.

The queue-drain calculation is a sizing check, not a throughput guarantee. It must use the encoded WebSocket-frame size, the capacity of the lane the frame actually occupies, and bytes already handed to TCP. At a measured 32 KiB/s drain rate, the default 15-second maximum sojourn, and an illustrative 64 KiB already ahead in the socket:

Frame and lane Capacity Drain rate Full-path drain Sojourn headroom
200-byte reliable data 1,024 163.8 msg/s 8.25 s 6.75 s
600-byte reliable data 1,024 54.6 msg/s 20.75 s -5.75 s
419-byte DeliveryReport control 128 78.2 msg/s 3.64 s 11.36 s

The 600-byte row is intentionally over budget: at that link rate, the default 1,024-slot data queue is burst/memory capacity, not a promise that a completely full queue can drain before the oldest item reaches 15 seconds. The corresponding maximum capacity is 709 messages after reserving that illustrative socket backlog, before additional WebSocket framing overhead. The configured send buffer is only a request and the kernel may clamp or account it differently, so measure the effective path rather than treating 709 as a portable safe value. Lower send_queue_capacity, increase measured egress, or classify replaceable data as latest/volatile; do not raise max_sojourn_ms without accepting the larger stale-data and room-freeze bound. Control reports use control_queue_capacity=128, not the data queue's 1,024 slots.

Batching latency

Batching is opt-in: websocket.enable_batching defaults to false, so latency-sensitive relay traffic (rollback game data is DeliveryClass::Reliable) is never held by the flush timer. Combined with TCP_NODELAY on every accepted socket (below), a sparse relay hop is bounded by network and local processing rather than a batch timer or Nagle stall — integration testing measured round-trip latency fall from ~46 ms to ~12-20 ms once default batching was removed.

When batching is enabled for bulk/throughput deployments, only DeliveryClass::Latest waits — up to websocket.batch_interval_ms (16 ms by default) — for a fuller batch so same-key values can coalesce; control, Reliable, and Volatile are still released immediately. A full batch_size flushes earlier. Enabling batching therefore adds at most one batch_interval_ms per hop, and only to Latest traffic. Raise the interval only after measuring the syscall/throughput tradeoff, keep it at or below the 60000 ms startup ceiling, and keep max_sojourn_ms > batch_interval_ms (both enforced at startup).

Accepted sockets disable Nagle (TCP_NODELAY) on both the plain (axum::serve) and TLS (axum_server) serve paths, so small bidirectional frames avoid the ~40-90 ms Nagle x delayed-ACK stall on loopback. A WebSocket sink flush does not disable Nagle; the option is set explicitly on each accepted stream.

Directional partition detection

A WebSocket can fail in only one direction. One-way application traffic is not proof that both directions are healthy. Successful server writes do, however, prove progress at the bounded server socket-write boundary, so they suppress or cancel a redundant Pong deadline while queue sojourn and later writes continue to own reverse-path stalls. The server has two independent fail-loud mechanisms:

Fault What may still work Default detection path Authoritative result
client → server blackhole The client can keep receiving room traffic and RFC 6455 Ping frames While application writes keep completing, Pings are still emitted but their Pong deadlines are superseded because output may be ahead of them; the inbound-activity reaper owns detection (server.ping_timeout, 30 seconds by default). Once outbound traffic is idle, the next protocol probe deadline also applies; its worst-case fixed-tick bound from that idle point is nearly 2 × server_ping_interval_secs + pong_timeout_secs (25 seconds by default) close 4003 activity_timeout; the activity-reaper or WebSocket-ping metric increments according to which mechanism wins
server → client blackhole, both directions otherwise idle No inbound non-Pong frame proves liveness The client never receives the next protocol Ping, so no matching Pong returns; the same worst-case fixed-tick bound applies close 4003 activity_timeout
server → client blackhole while client → server traffic continues Client writes can still reach the server, so inbound activity skips or cancels idle-only probes Outbound queue/socket pressure, a socket write failure, or an application-level policy; upstream traffic alone cannot prove the reverse path close 4002 slow_consumer when delivery pressure wins; otherwise detection requires another policy
symmetric blackhole Neither direction carries new traffic The next protocol probe cannot complete close 4003 activity_timeout

The close code describes the mechanism that won, not the physical direction of the fault. Idle-only probes deliberately treat decoded inbound non-Pong traffic and completed outbound application writes as progress. This prevents a Pong queued behind traffic on a constrained but draining connection from being misclassified as a transport timeout. Continuous client → server traffic does not independently test the reverse path; outbound pressure/failure or an application policy owns that case. Keep protocol Ping handling enabled, treat 4002 and 4003 as terminal for that physical connection, and reconnect according to the client contract.

The converse also matters: continuous server → client application progress does not prove the client → server path. Because that progress can legitimately defer a Pong deadline behind constrained output, the default server.ping_timeout inbound-activity reaper supplies the independent 30-second bound. Setting it to 0 disables that reaper; with continuous outbound progress, a client → server half-partition is then left to TCP failure or an application policy rather than a fixed server deadline. Keep the reaper nonzero when that bound is required.

The directional real-socket experiments use shortened probe/grace values and prove all three distinct cases: the unaffected direction carries traffic, exactly the expected close metric fires, the surviving member observes PlayerLeft, and a replacement member can join and receive relayed data. The same suite confirms that E4's probe bound supersedes the older activity-reaper estimate for these socket partitions.