Operator Handbook

This handbook is the checklist for running a game on the supported single-machine Compose stack. The exact commands and settings are in the Deployment Reference, and what each kind of update does to players’ connections is in Connections During Updates. Run every command from the deploy/ folder, with the same Compose file that created the stack.

First claim and deployment

Choose how the web side reaches the internet:

• docker-compose.prod.yml includes Caddy. Open web ports 80 and 443 and the telnet port you intend to use; keep the game server, NATS, metrics and connection-server web ports private.
• docker-compose.cloudflare.yml uses a Cloudflare Tunnel for the web side and hides the web origin. Telnet still needs its own, deliberate public route, a DNS-only record that shows the machine’s address.

Create .env from .env.example, set unique secrets, restrict who can read the file, and keep it out of source control and anything you share for troubleshooting. Set up restic only if you turn backups on, then start the stack.

Danger

Protect /setup from untrusted visitors, then claim it immediately. The first visitor owns the pre-generated administrator account linked to #1.

Right after the claim:

1. Log in as the administrator in a fresh browser, log in a disposable test character over plaintext telnet (never reuse a production password for this check), and log in through the portal terminal at wss://your-domain/ws.
2. Confirm /ws reaches the connection server and the health and readiness checks are green.
3. Read the container logs for database, NATS, renderer, login and proxy errors, without sharing tokens or .env values.
4. Scrape the private Prometheus metrics, and set alerts for availability, disk headroom, queue pressure and repeated failed logins.
5. Run @storage as a wizard and write down the map limit, live data, disk allocated, what a backup needs at its peak, free space, and any leftover worlds.
6. Make your own staff character and keep #1 for emergencies. God passes every lock and permission check, so a mistake made as God has nothing to stop it. Create a character on your account, then, connected as #1, type @set *<name>=WIZARD. The portal’s first-players checklist, at /admin/onboarding, goes through this and the rest of opening your game.

What recovery actually covers

World copies

@backup makes a point-in-time copy of the world. Never let a file backup read the live data.mdb.

Wiki files

Uploaded files live outside the world and must be included separately in restic snapshots.

NATS data

NATS’s stored data lets browsers recover after a restart. It is not a backup of the world.

Plugins and settings

Keep plugin files and deployment settings. Keep .env values in a separate secrets store.

These are not interchangeable:

What	Where it lives	How to protect it
The world	sharpmush-server’s app-data volume	Use @backup to make a point-in-time copy. Never let restic or another tool read the live data.mdb.
Wiki files	app-data volume, outside the world	Include them directly in restic snapshots. A world copy does not contain uploads.
NATS data	The nats-data volume	Keep it so browser sessions can recover across restarts, and back it up if that matters to you. Losing it never means losing the world.
Key ring	app-data volume, lightning.dataprotection-keys beside the world	Back it up (the restic container does), apart from the world copies where you can. Without it the world’s secrets must be set again.
Installed plugins	app-data volume, lightning.plugins beside the world	Included in restic snapshots. Restore the whole folder with the world.
Deployment settings	The deploy/ folder and .env	Keep them in version control or snapshots, with .env values in a separate secrets store.
The restic repository	Off-machine storage	Protect its password separately, watch the nightly results, and test restores. It holds safe world copies and the other files you chose, not a live snapshot of the running database.

Keeping backups is not the same as knowing they work. Check the nightly results and @backup/list, and also run a restore drill on a schedule.

Restore drill

1. Pick a dated restic snapshot and restore it to a separate folder or machine. Never overwrite the live volume during a drill.
2. Check the restored set has a finished world copy, the wiki files, the plugins you need and your deployment settings. Don’t treat lock.mdb as data.
3. Start a private stack on the restored files with new, test-only secrets, and no production DNS, mail, bots or outgoing automation.
4. Check the world is the right one: known dbrefs, administrator and ordinary player login, important softcode, mail and channels, wiki pages and files, plugins, telnet, and the browser terminal.
5. Write down the snapshot id, date, who ran the drill, how long it took, what was recovered, the checks, any failures and what was done about them. A drill is finished only when the record names a recovery point that works.

The restore commands are in the Deployment Reference.

Routine checks

• Readiness, container restart counts, recent warnings and errors, metrics alerts, free disk space, and @storage.
• Backups finished, and the off-machine copies are being kept as intended.
• Leftover .previous, .precompact, .staging-* and unfinished backup folders. Remove them only after whatever replaced them is checked (see Earlier worlds).
• Certificate expiry, proxy routes, that telnet answers, and a real portal login (a page loading is not enough).
• Changes to accounts, bans, site locks and administrators. Replace any exposed secret and sign out affected sessions promptly.

Upgrade and rollback

Keep the image tags and automatic updates the deployment already uses. Before a deliberate upgrade: read the release notes and Connections During Updates, check there is disk room, make and check a fresh world copy, and write down the current image digests and Compose settings. Then pull the images, replace only the services you meant to, and repeat the readiness, logs, metrics, storage, login, command, portal and integration checks.

Replacing the game server or the renderer keeps players connected. Replacing the connection server closes every connection: logged-in browsers can recover within their recovery window, and telnet players reconnect. Losing the machine, its disk or NATS’s data has wider effects. Schedule accordingly.

Rolling back a deployment means going back to the previous image and Compose settings, when the data still works with them. Restoring data means stopping changes and replacing the stored world from a checked recovery point. Neither replaces the other. Don’t start an older version against a world a newer version has already upgraded, unless that downgrade is stated to be supported.

When something goes wrong

Symptom	Check first	Protect first
MDB_MAP_FULL, low disk, or a fast-growing world	@storage, free disk, leftover worlds, what a backup needs	Stop unnecessary changes, and never delete the only recovery copy. Raise the map limit before trying again.
NATS down, or browser recovery failing	NATS health and logs, its volume, retention and message size limits	Keep its volume. Expect coordination between the parts of the game, and recovery, to suffer.
Output delayed, but players still connected	Renderer health and the permissions on its socket	Keep the connection server running and replace only the renderer.
Every telnet and browser connection drops	The connection server’s history, the machine and network, the proxy and recent deployments	Stop recreating it over and over; find out whether browsers can recover or players must log in again.
Portal login or setup fails	Server readiness, the proxy’s host and HTTPS headers, sessions, site locks, bans	Keep /setup protected, and don’t turn off login protections to diagnose.
Suspected break-in	Account and audit records, exposed secrets, sessions, images and machine access	Cut off public access if needed, keep the logs, replace secrets, sign out sessions, and restore only from data you trust.

When collecting information to share, include times, image digests, service state, relevant health and metrics readings, and short log excerpts. Remove passwords, restic credentials, cookies, tokens, connection strings and the contents of .env.

Deployment Reference Compose files, ports, backups, disk space, restore commands, and Cloudflare setup.

Deployment architecture What each part of the game owns, how they talk, and what a restart of each one does.

Optional analysis transports

The boot-time MCP and LSP plugins are independently optional and disabled by default. Their authentication and rate limiting belong to the host. See Remote MCP and LSP Analysis for setup, editor usage, isolation and resource limits. Local validation and remaining real-network checks are documented there.