Skip to content
Chatto v0.5.0

Server Operations

To operate Chatto, keep the process healthy, monitor storage growth, test your backups, and know which tool repairs an account or a deployment.

SurfaceUse for
Admin UIMembers, roles, permissions, room layout, branding, event log, and system diagnostics.
Metrics listenerPrometheus metrics for one server process.
ExporterPrometheus metrics for the full deployment, from NATS and asset storage.
Operator CLILocal recovery and bootstrap over a Unix socket.
BackupsDisaster recovery, migration tests, and upgrade safety.

Chatto currently has no supported operation to delete a room’s event or message data. Archiving only hides a room from active room lists. It does not erase the room’s data. Deleting and recreating a room is not a supported way to clear its data. Do not rely on creating a new room with the same name.

Account deletion has specific cleanup behavior. See Encryption at Rest and Data Erasure for the data that account deletion removes or makes unreadable, and the event metadata that remains.

Before deployment, run the check with the same environment as the server:

Terminal window
chatto config check --config /etc/chatto/chatto.toml

The command returns a nonzero exit status for unknown or deprecated TOML keys or CHATTO_ environment names. It then checks decoding and normal configuration validation. It does not start the server or contact external services. A successful check does not test network access, credentials with a provider, or storage access.

The check includes nested tables, repeated provider tables, fixed ENV names, and indexed provider and development bootstrap ENV names. Indexes must start at 0 and have no gaps. Names outside CHATTO_ are ignored. Run the check in the server environment: development-task or test variables with that prefix are not server configuration and are reported as unknown.

Diagnostics show key names and migration instructions, never setting values. Syntax, decoding, and value-validation errors omit details that could contain secrets or personal data. Check required fields and value formats in the configuration reference if you get a value error.

Without --config, the command uses chatto.toml in the current directory. If that file is absent, it checks ENV-only configuration. An explicitly selected file must exist.

Normal startup keeps its upgrade-compatible loading policy. It still ignores unknown TOML keys and ordinary unknown ENV names. Supported aliases still work, but the explicit check rejects them so you can migrate before deployment:

Old settingAction
livekit.instance_id, CHATTO_LIVEKIT_INSTANCE_IDUse livekit.server_id, CHATTO_LIVEKIT_SERVER_ID.
bootstrap.instanceUse bootstrap.server.
bootstrap.users[].instance_roleUse bootstrap.users[].server_role.
CHATTO_AUTH_OIDC_*Use the indexed provider variables.
operator_api.socket_mode, CHATTO_OPERATOR_API_SOCKET_MODERemove the setting. The socket always uses mode 0600.
livekit.webhook_url, CHATTO_LIVEKIT_WEBHOOK_URLRemove the unused setting. Set webhook.urls in LiveKit.
auth.providers[].provider_optionsRemove the unused setting. Use documented provider fields.

Email OTP codes use a 30m default in generated, omitted-setting, and ENV-only configuration. Explicit TOML values remain valid, and CHATTO_AUTH_EMAIL_OTP_TTL overrides them.

Owners and administrators use the admin UI for daily operations. Their permissions control which pages they see. The System page has tabs. The Overview tab starts with a health summary. It shows a lost broker connection, failed projections, stalled background workers, and JetStream usage near an account limit. The Streams, Projections, Workers, and LiveKit tabs show the details. Some checks link to the tab with their details. A dot on a tab shows a warning or a problem.

The system pages show operational metadata, not message content. This is an application boundary. A person who controls the process, the storage, and the keys can read more. See Security & Privacy.

Each server process creates its own NATS consumers to keep local read models. Their names start with projection-. Chatto deletes them at shutdown. After a crash, NATS removes them after five minutes without activity. Do not delete named durable consumers, such as chatto-asset-cleanup-v1. They keep unfinished background work across restarts.

Enable the metrics listener to let Prometheus scrape one server process:

[metrics]
enabled = true
bind_address = "127.0.0.1"
port = 9090
path = "/metrics"

Keep the metrics listener private. Enable metrics.pprof only while you profile, and only on localhost or a trusted internal network.

Use these metrics to check snapshot restore, publication, and cleanup. The metrics listener reports each replica separately. Counters reset on restart.

MetricTypeLabels
chatto_projection_snapshot_restore_infoGaugeprojection, source, reason
chatto_projection_snapshot_publications_totalCounterprojection, backend, result
chatto_projection_snapshot_publication_duration_secondsHistogramprojection, backend
chatto_projection_snapshot_last_publication_timestamp_secondsGaugeprojection, backend
chatto_projection_snapshot_cleanup_runs_totalCounterbackend, result
chatto_projection_snapshot_cleanup_deleted_objects_totalCounterbackend
chatto_projection_snapshot_cleanup_deleted_bytes_totalCounterbackend
chatto_projection_snapshot_cleanup_last_success_timestamp_secondsGaugebackend

The restore gauge has value 1 after initial replay completes. Its source is current, previous, or cold. Its reason is none, disabled, missing, rejected, or unavailable. disabled also applies to projections that always use cold replay. For previous, the reason describes why the current generation was not used. If both generations fail, unavailable takes precedence over rejected, and rejected over missing.

Compare the restore result with chatto_projection_startup_duration_seconds and chatto_projection_startup_messages for the same replica and projection. The duration and message count describe that process’s initial replay.

Publication and cleanup results are success or error. A skipped publication or a lease held by another replica does not count as an attempt. Publication duration includes capture, encoding, and storage work for completed attempts, including failures. Cleanup deletion counters include work completed before a cleanup failure. NATS uses Object Store TTL, so these cleanup metrics measure Chatto’s S3 cleanup only.

Operation metrics are absent until the first completed attempt. Success timestamps are absent until this replica completes successful work. They do not advance when another replica publishes or cleans up. To check the latest observed success across replicas, take the maximum timestamp within one server deployment. A restarted replica does not retain its previous timestamps.

For example, this query selects replicas that used cold replay because snapshot storage was unavailable:

chatto_projection_snapshot_restore_info{source="cold",reason="unavailable"} == 1

The exporter gives metrics for the full deployment. On a single process, run it inside chatto run:

[exporter]
enabled = true
bind_address = "127.0.0.1"
port = 9100
path = "/metrics"

With more than one replica, enable the embedded exporter on one replica only. Alternatively, run chatto exporter as a separate process against external NATS or the TCP listener of embedded NATS.

For S3 storage, the exporter reports chatto_projection_snapshot_storage_objects and chatto_projection_snapshot_storage_bytes. Both have backend="s3" and a scope label: current, non_current, or all_versions. These scopes refer to S3 object versions, not the current and previous snapshot generations. Object counts for all_versions include delete markers; markers add no bytes.

The totals cover the private snapshot namespace under the configured S3 path prefix, including old contracts and encryption epochs. They exclude assets. The exporter counts keys without downloading or decrypting objects. These listings are periodic; totals can change while a listing is in progress.

chatto_projection_snapshot_storage_refresh_success{backend="s3"} is 1 only when the latest inventory completed. It is 0 before the first inventory or after a failure; snapshot totals are absent in those cases. Use chatto_s3_bucket_last_refresh_timestamp_seconds and chatto_s3_bucket_refresh_duration_seconds to check the shared inventory’s attempt time and duration. Reading metrics does not start an S3 listing.

Schedule encrypted backups, keep them away from the server’s data volume, and test restores. See Backup & Restore for key exports, S3 assets, retention, and disaster recovery.

The search index contains text from decrypted messages. Keep it on a protected local volume and do not include it in backups. Chatto rebuilds a missing index. See Enabling Search for the configuration, separate provider processes, and index recovery.

Enable the operator API only when you need local recovery or trusted automation:

[operator_api]
enabled = true
socket_path = "/tmp/chatto/operator.sock"

Run chatto operator as the same OS user as the Chatto process. In a container, use docker compose exec as the chatto user. Use it to:

  • list users when no admin session is available
  • set a password for a user
  • create a bootstrap user
  • add a verified email address
  • assign or revoke a role
  • delete an account

See Operator CLI for all commands.

  1. Read the release notes.
  2. Make a new backup.
  3. On a large server, restore a copy into a test environment and start the new version against it.
  4. After startup, examine the system diagnostics and projection readiness.
  5. Monitor logs and metrics for the first minutes.
  6. If you operate bots or custom clients, read the API changes.

To operate more than one replica, see Horizontal Scaling.

Public badges expose the server’s registered and online member counts. They are off by default. Enable them only if you intend to publish this activity. See the Shields configuration reference.