Architecture¶

System design and project structure overview.

System Overview¶

Text Only

┌─────────────────────────────────────────────────────┐
│              Game Clients (WebSocket)                │
│  Browser | Unity | Godot | Custom                    │
└────────────────────┬────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────────┐
│            Signal Fish Server (Rust)                 │
│  ┌───────────────────────────────────────────────┐  │
│  │  WebSocket Layer (axum)                       │  │
│  │  - Upgrade handling                           │  │
│  │  - Message batching                           │  │
│  │  - Auth enforcement                           │  │
│  └───────────────┬───────────────────────────────┘  │
│                  ▼                                   │
│  ┌───────────────────────────────────────────────┐  │
│  │  EnhancedGameServer (Core Logic)              │  │
│  │  - Room management                            │  │
│  │  - Player state                               │  │
│  │  - Authority management                       │  │
│  │  - Message routing                            │  │
│  └───────────────┬───────────────────────────────┘  │
│                  ▼                                   │
│  ┌───────────────────────────────────────────────┐  │
│  │  In-Memory Storage                            │  │
│  │  - Rooms (DashMap)                            │  │
│  │  - Players (DashMap)                          │  │
│  │  - Rate limits                                │  │
│  │  - Reconnection tokens                        │  │
│  └───────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────┘

Core Components¶

WebSocket Layer¶

Location: src/websocket/

Handles WebSocket upgrade, message serialization, and connection lifecycle.

Key modules:

handler.rs - WebSocket upgrade and initialization
connection.rs - Connection lifecycle management
batching.rs - Outbound message batching
sending.rs - Message serialization (JSON/MessagePack)
routes.rs - Axum router configuration

EnhancedGameServer¶

Location: src/server.rs

Core server logic coordinating all operations.

Responsibilities:

Room creation and lifecycle
Player join/leave handling
Message routing between players
Authority management
Lobby state transitions
Cleanup tasks

Protocol Layer¶

Location: src/protocol/

Defines message types, validation, and domain types.

Key modules:

messages.rs - ClientMessage and ServerMessage enums
types.rs - PlayerId, RoomId, domain types
room_state.rs - Room, Player, LobbyState
validation.rs - Input validation functions
room_codes.rs - Room code generation
error_codes.rs - ErrorCode enum

Protocol v3 Session Layer (Topology + Transport)¶

Location: src/server/session_policy.rs, src/server/signaling.rs

On /v3/ws (the default v3 endpoint), the server negotiates a peer-to-peer session plan instead of pure relay: a two-axis ladder (mesh+webrtc -> host+webrtc -> host+direct -> relay) with a per-recipient SessionPlan and Signal messages for WebRTC signaling brokering and host-failover / membership re-planning. A single v2 or relay-only member forces the whole room to the relay floor; each v3 member receives an explicit no-peer relay plan while v2 behavior stays byte-identical. The server also mints ephemeral coturn-REST TURN credentials for operator-configured external TURN URLs and advertises STUN; optional ice_servers are attached to RoomJoined / Reconnected for v3 WebRTC-capable clients when ICE pre-gather is enabled.

See:

Configuration¶

Location: src/config/

Configuration loading, validation, and defaults.

Structure:

loader.rs - JSON file + environment variable loading
validation.rs - Config validation rules
defaults.rs - Default values
Type-specific configs: server.rs, security.rs, protocol.rs, etc.

Database¶

Location: src/database/

Storage abstraction with in-memory implementation.

GameDatabase trait - Abstract storage interface
InMemoryDatabase - Default implementation using DashMap

Designed for extension with custom backends (Redis, PostgreSQL, etc.).

Authentication¶

Location: src/auth/

App-based authentication and per-app rate limiting.

middleware.rs - InMemoryAuthBackend
rate_limiter.rs - Per-app token bucket rate limiter
error.rs - Auth error types

Coordination¶

Location: src/coordination/

In-memory coordination primitives and extension seams. The shipped implementations do not coordinate multiple server processes; see the single-instance deployment contract.

room_coordinator.rs - Room operation coordination
dedup.rs - Deduplication cache (LRU)

Metrics¶

Location: src/metrics.rs, src/websocket/metrics.rs, src/websocket/prometheus.rs

Metrics collection and export.

Atomic counters for room/player counts
HDR histograms for latency tracking
JSON and Prometheus export formats

Data Flow¶

JoinRoom Resolution¶

JoinRoom is the only room-entry message. EnhancedGameServer resolves each request by (game_name, room_code):

Validate/rate-limit the request.
If room_code is omitted, generate one and create a new room.
If room_code is provided:
If room exists for that game_name, add player to existing room.
If room does not exist for that game_name, create a new room with the requested room_code.
Send RoomJoined to the caller. If they joined an existing room, broadcast PlayerJoined to other room members.

Text Only

Client                    WebSocket Handler         EnhancedGameServer        Database
  |                              |                         |                      |
  |-- JoinRoom ----------------->|                         |                      |
  |                              |-- handle_message ------>|                      |
  |                              |                         |-- get_room --------->|
  |                              |                         |<-- hit or miss ------|
  |                              |                         |-- add_player/create->|
  |                              |<-- RoomJoined ----------|                      |
  |<-- RoomJoined ---------------|                         |                      |
  |                              |                         |                      |
  |<-- PlayerJoined -------------|<-- broadcast (join only)                       |
(other clients)                                                                   |

Game Data Relay¶

Text Only

Client A                 EnhancedGameServer                    Client B
  |                              |                                  |
  |-- GameData ----------------->|                                  |
  |                              |-- broadcast to room ------------->|
  |                              |                                  |<-- GameData

Concurrency Model¶

Async Runtime¶

Uses Tokio for async I/O and task scheduling.

Shared State¶

Arc<DashMap> for concurrent room/player access
RwLock for infrequent writes (config, auth state)
Atomic counters for metrics

Message Passing¶

Bounded per-connection data and control queues
Class-aware data admission: reliable backpressure, loss-accounted v3 latest/volatile
Generation-scoped v3 control priority for peer lifecycle and exact accountability

Locking Strategy¶

Fine-grained locking per room
Never hold locks across .await points
Use tokio::spawn to avoid blocking

Scalability¶

Single Instance Limits¶

Recommended per instance:

Rooms: < 500 active rooms
Players: < 2000 concurrent players
Messages: < 10,000 messages/second

Multi-Instance Deployment¶

Transparent multi-instance deployment is not supported. RoomOperationCoordinator and DistributedLock use in-memory implementations and cannot preserve a room across processes. Run one active process per routing domain, or use an external directory that assigns isolated room homes before WebSocket connection. See Single-Instance Deployment.

Storage Options¶

In-Memory (default) - Fast, ephemeral, no external dependencies
Custom persistent backends would require a separately designed distributed authority and routing protocol; implementing GameDatabase alone is not sufficient for multi-instance safety.

Security Architecture¶

Layers¶

Transport - Optional TLS (feature-gated)
Authentication - App-based auth (optional)
Rate Limiting - Per-IP and per-app limits
Validation - Input validation at protocol boundaries
Connection Limits - Max connections per IP

Trust Model¶

Untrusted clients - All input validated
Semi-trusted apps - App authentication + rate limits
Trusted admin - Metrics endpoints (optional auth)

Module Dependencies¶

Text Only

main.rs
  └── lib.rs
       ├── server.rs (EnhancedGameServer)
       │    ├── server/admin.rs
       │    ├── server/authority.rs
       │    ├── server/connection_manager.rs
       │    ├── server/dashboard_cache.rs
       │    ├── server/game_data.rs
       │    ├── server/heartbeat.rs
       │    ├── server/maintenance.rs
       │    ├── server/message_router.rs
       │    ├── server/messaging.rs
       │    ├── server/ready_state.rs
       │    ├── server/reconnection_service.rs
       │    ├── server/relay_policy.rs
       │    ├── server/room_service.rs
       │    ├── server/session_policy.rs (Protocol v3 session-plan selection/emission/re-planning)
       │    ├── server/signaling.rs (Protocol v3 signal relay + finalized membership refresh)
       │    ├── server/spectator_handlers.rs
       │    └── server/spectator_service.rs
       ├── protocol/
       │    ├── messages.rs (ClientMessage, ServerMessage)
       │    ├── types.rs (PlayerId, RoomId, etc.)
       │    ├── room_state.rs (Room, Player, LobbyState)
       │    ├── room_codes.rs (Room code generation)
       │    ├── validation.rs (Input validation)
       │    └── error_codes.rs (ErrorCode enum)
       ├── database/
       │    └── mod.rs (GameDatabase trait, InMemoryDatabase)
       ├── coordination/
       │    ├── mod.rs (MessageCoordinator trait)
       │    ├── room_coordinator.rs (RoomOperationCoordinator)
       │    └── dedup.rs (DedupCache)
       ├── auth/
       │    ├── middleware.rs (InMemoryAuthBackend)
       │    ├── rate_limiter.rs (Per-app rate limiter)
       │    └── error.rs (AuthError types)
       ├── websocket/
       │    ├── handler.rs (WebSocket upgrade)
       │    ├── connection.rs (Connection lifecycle)
       │    ├── batching.rs (Message batching)
       │    ├── sending.rs (Serialization + send)
       │    ├── token_binding.rs (Token binding)
       │    ├── routes.rs (Axum router)
       │    ├── metrics.rs (/metrics endpoint)
       │    └── prometheus.rs (Prometheus format)
       ├── config/
       │    ├── types.rs (Root Config struct)
       │    ├── server.rs (ServerConfig)
       │    ├── protocol.rs (ProtocolConfig)
       │    ├── security.rs (SecurityConfig)
       │    ├── websocket.rs (WebSocketConfig)
       │    ├── logging.rs (LoggingConfig)
       │    ├── relay.rs (RelayTypeConfig)
       │    ├── session.rs (SessionConfig: topology/transport policy, ICE pre-gather)
       │    ├── turn.rs (TurnConfig: ephemeral coturn-REST TURN credentials + STUN)
       │    ├── ice_url.rs (ICE URL scheme validation)
       │    ├── coordination.rs (CoordinationConfig)
       │    ├── metrics.rs (MetricsConfig)
       │    ├── defaults.rs (Default values)
       │    ├── loader.rs (JSON + env loading)
       │    └── validation.rs (Config validation)
       ├── security/
       │    ├── tls.rs (TLS support, feature-gated)
       │    ├── crypto.rs (AES-GCM envelope encryption)
       │    ├── token_binding.rs (Channel-bound tokens)
       │    └── turn_credentials.rs (ephemeral coturn-REST TURN credential minting)
       ├── metrics.rs (AtomicU64 + HDR histograms)
       ├── logging.rs (Structured logging init)
       ├── distributed.rs (InMemoryDistributedLock)
       ├── broadcast.rs (Zero-copy broadcast primitives)
       ├── rate_limit.rs (In-memory RoomRateLimiter)
       ├── reconnection.rs (In-memory ReconnectionManager)
       ├── retry.rs (Exponential backoff utility)
       └── rkyv_utils.rs (Zero-copy serialization helpers)

Architecture Decision Records¶

Key architectural decisions are documented in Architecture Decision Records (ADRs).

Current ADRs:

ADR-001: Reconnection Protocol - WebSocket reconnection and event replay
ADR-0001: Protocol v3 Two-Axis (Topology + Transport) - Capability-gated signaling
ADR-0003: Formal Verification and Fuzzing - Verifying the v3 session core

Design Principles¶

Zero-Cost Abstractions¶

Prefer value types over heap allocations
Use Bytes for network data (zero-copy)
Borrow instead of clone where possible

Fail-Fast Validation¶

Validate at system boundaries
Return typed errors (Result<T, E>)
Never .unwrap() in production code

Graceful Degradation¶

Handle partial failures
Make delivery policy explicit: reliable data waits on the bounded data queue and closes a slow recipient loudly; v3 latest and volatile data never backpressure and record every omission in an exact prior DeliveryReport
Keep v3 control traffic on a bounded lane that has strict priority within the active recipient generation; use barriers for the recipient's own room and spectator transitions. If the server cannot publish exact accountability before later data, or the oldest outbound item cannot complete its socket write within websocket.max_sojourn_ms, fail closed with 4002 slow_consumer
Timeout on slow operations

Observable by Default¶

Structured logging with tracing
Metrics for all operations
Correlation IDs for request tracking

Testing Strategy¶

Unit Tests¶

Module-level tests in the same file:

Rust

#[cfg(test)]
mod tests {
    #[test]
    fn test_room_code_validation() {
        // ...
    }
}

Integration Tests¶

Multi-component tests in tests/:

Rust

#[tokio::test]
async fn test_room_lifecycle() {
    // ...
}

E2E Tests¶

Full WebSocket session tests:

Rust

#[tokio::test]
async fn test_websocket_flow() {
    // ...
}

Next Steps¶

Development - Building and testing
Library Usage - Embedding the server
Protocol Reference - Message types