Production Deploy Runbook

For operator phone-to-Railspike intake, see deployment and Codex worker configuration.

This runbook describes the intended production deployment model for the PHP forum rewrite.

Deployment Model

The intended production shape is:

• Apache serves public/ as the DocumentRoot

• PHP handles dynamic requests through public/index.php

• Apache directly serves only existing /assets/* files and favicon.ico

• the PHP front controller serves eligible pages from an atomically selected static release

• the canonical writable repository lives outside public/

• derived state under state/ is writable by the web user

This application assumes a conservative shared-host style environment with:

• PHP 8.1+

• PDO SQLite

• mbstring

• standard filesystem functions

• shell access sufficient for non-interactive git commands

• Apache with mod_rewrite

Documentation uses the project-owned renderer; no third-party Markdown bundle or Composer step is required. After renderer updates, refresh generated Docs pages and verify the reported document. See Markdown rendering, regeneration and rollback.

Required Directory Layout

One workable layout:

/srv/forum-rewrite/
  app/                      application checkout
    public/
    src/
    templates/
    scripts/
  repository/               writable canonical content checkout
    records/
    .git/
  state/
    cache/
    private/
    static_html/
      releases/
      current -> releases/release-...   active forum HTML release
    static_html-docs/
      releases/
      current -> releases/docs-...      active complete docs library
    forum-rewrite.lock
    read_model_stale.json

Recommended mapping:

• application root: /srv/forum-rewrite/app

• Apache DocumentRoot: /srv/forum-rewrite/app/public

• writable repository root: /srv/forum-rewrite/repository

• read-model database: /srv/forum-rewrite/state/cache/post_index.sqlite3

• static release root: /srv/forum-rewrite/state/static_html

Optional runtime setting:

• FORUM_EXECUTION_LOCK_TIMEOUT_SECONDS: seconds a request waits for the shared write/read-model lock before returning a busy error. The default is 5. It may also be set in the private config file (see Deployment Settings); an environment value takes precedence. Prefer the private config file over SetEnv in public/.htaccess, which is tracked and blocks git pull when edited.

• FORUM_UNICODE_AUTHORED_TEXT: when set to true, subject/body prose may contain visible UTF-8 text such as Cyrillic. The default is disabled.

• FORUM_APP_VERSION_NOTIFICATION: when set to false, disables browser-side app version polling and the reload banner. The default is enabled.

• LLM_CONVERSATION_RECORDING_ENABLED: controls private exact-prompt/response capture. The default is enabled.

• LLM_CONVERSATION_UI_ENABLED: controls approved-user/operator web visibility of captured exchanges. The default is enabled.

• LLM_EXCHANGE_DATABASE_PATH: optional private SQLite path; defaults to <application-root>/state/private/llm_exchanges.sqlite3.

• VISITOR_STATISTICS_DATABASE_PATH: optional private SQLite path for aggregate visitor statistics; defaults to <application-root>/state/private/visitor_statistics.sqlite3.

• FORUM_TASK_QUEUE_DATABASE_PATH: optional private SQLite path for queued internal maintenance; defaults to <application-root>/state/private/internal_tasks.sqlite3.

• FORUM_TASK_QUEUE_EMERGENCY_LAUNCH_ENABLED: optional true enables one detached, allowlisted queue-worker launch when a classified schema recovery finds no fresh executor heartbeat. Leave it unset on hosts that prohibit child processes.

• FAST_SCORING_DATABASE_PATH: optional private SQLite score-state path; defaults to <application-root>/state/private/fast_scores.sqlite3.

• FAST_SCORING_AUTOMATIC_ENQUEUE_ENABLED: when true alongside FAST_SCORING_ENABLED, creates private score work for newly published posts only; defaults to false.

• PRIVATE_MESSAGE_DATABASE_PATH: optional private SQLite mailbox path; defaults to <application-root>/state/private/messages.sqlite3.

• PRIVATE_MESSAGE_HISTORY_SYNC_DATABASE_PATH: separate private history-transfer SQLite path; defaults to <application-root>/state/private/message_history_sync.sqlite3. Must resolve to a different file from the mailbox.

Writable Paths

The web user must be able to write:

• the canonical repository checkout at FORUM_REPOSITORY_ROOT

• the parent directory of FORUM_DATABASE_PATH

• the lock file directory next to FORUM_DATABASE_PATH

• FORUM_STATIC_HTML_ROOT and its releases/ directory

• its -docs sibling and releases/ directory for the docs publisher

• state/private/agent-reply/ under the application root if agent reply fulfillment is enabled

• the parent directory of LLM_EXCHANGE_DATABASE_PATH if LLM conversation recording is enabled

• the parent directory of VISITOR_STATISTICS_DATABASE_PATH when visitor statistics are in use

• the parent directory of FORUM_TASK_QUEUE_DATABASE_PATH when the internal task queue is enabled

• the parent directory of FAST_SCORING_DATABASE_PATH when Fastmod sweeps are enabled

• the parent directories of PRIVATE_MESSAGE_DATABASE_PATH and PRIVATE_MESSAGE_HISTORY_SYNC_DATABASE_PATH when private messaging is enabled

Static HTML is derived state. A write removes the current release pointer, so subsequent public requests use PHP until a fresh complete release is published. Old release directories are retained and are never edited in place. Docs have a separate lock and pointer; forum writes, imports, builds and cleanup do not render or bulk scan/copy docs. A small legacy reference retains docs from an old combined release until independent publication succeeds.

Deployment Settings

Six settings tell a deployment which site it is and where its data lives:

• FORUM_SITE_ID: the site profile (zenmemes, chouse, qdb or mitrapclub); selects branding, default theme and composer copy, not content

• FORUM_REPOSITORY_ROOT: the canonical repository checkout

• FORUM_DATABASE_PATH: the read-model SQLite database

• FORUM_STATIC_HTML_ROOT: static releases; defaults to <application-root>/state/static_html (with a profile suffix for non-ZenMemes profiles)

• FORUM_PUBLIC_ARTIFACT_ROOT: the public web root; defaults to <application-root>/public

• FORUM_IMAGE_ROOT: private image storage; see Image backup and recovery

Put them in the checkout's private config file, outside the code checkout. The web app, every ./v3 command, cron and cache workers then read the same values, and git reset --hard or git pull cannot remove them:

<?php
return [
    'FORUM_SITE_ID' => 'qdb',
    'FORUM_REPOSITORY_ROOT' => '/home/qdb/v3-data/repository',
    'FORUM_DATABASE_PATH' => '/home/qdb/v3-data/state/post_index.sqlite3',
    // ...existing LLM and feature settings...
];

Paths must be absolute. For each setting the order is: environment variable, then the private file, then the built-in default. Command options such as --repository-root still win for that command. Do not put these settings in SetEnv lines in the tracked public/.htaccess: a code reset discards them, and cron never sees Apache's SetEnv values.

Which private file. FORUM_SECRETS_PATH names it explicitly. Otherwise, a checkout at ~/sites/qdb uses ~/sites/forum-private/qdb/secrets.php when that file exists, else the shared ~/sites/forum-private/secrets.php. Two sites whose checkouts share a parent folder therefore each get their own file: create forum-private/<checkout folder name>/secrets.php for each. Renaming or re-cloning a checkout under another folder name switches it back to the shared file; keep the folder name or set FORUM_SECRETS_PATH. Edit the file directly or with ./v3 private-config edit; both use the same location.

Check. ./v3 status ends with a Deployment settings section listing the private file in use and every setting's value and source (environment, private file <path>, a command option, or default). It warns when the tracked public/.htaccess or public/index.php has local edits. ./v3 task-queue cron prints the file the cron worker reads.

Move existing SetEnv lines. For each SetEnv FORUM_... line in public/.htaccess or the vhost:

1. Add the same key and value to the private file.

2. Run ./v3 status and confirm it shows the value with (private file ...).

3. Remove the SetEnv line, restore the tracked .htaccess with git checkout -- public/.htaccess, and reload the site.

4. Run ./v3 status again: no .htaccess warning, and every value from the private file.

Vhost-level SetEnv keeps working and still takes precedence; remove it once the private file is in place so web and cron cannot drift apart.

Recovery. If the site shows the wrong profile or data after an update, run ./v3 status. A value showing default means the private file is missing, misnamed or in the wrong folder; restore it from backup or fix the path shown. An unknown FORUM_SITE_ID in the private file is an error naming the file and the valid profiles. Removing these keys from the private file returns to the built-in defaults. In an emergency an environment variable still overrides the file.

QDB acceptance (operator, on the QDB host). After deploying this release:

cd ~/v3
./v3 status                       # FORUM_SITE_ID: qdb (private file ...), no .htaccess warning
git reset --hard origin/main      # simulate an ordinary code reset
./v3 status                       # unchanged
curl -s https://qdb.us/ | grep -o '<title>[^<]*</title>'   # QDB's own Welcome page title

Then confirm the next cron run's log (state/private/task_queue_cron.log) shows no profile or repository errors. Record the result in the deployment configuration plan.

For local development of another profile, FORUM_SITE_ID=chouse ./v3 start still works. Omitting FORUM_REPOSITORY_ROOT and FORUM_DATABASE_PATH uses the same default local instance state for every profile; only the disposable static HTML cache stays profile-specific.

Content routing and image storage

The front controller evaluates membership and image feature flags before serving content, including QDB hot paths. The checked-in .htaccess no longer serves QDB artifacts directly through public/.static; remove custom direct-content shortcuts. Matching generated HTML still uses the PHP static fast path.

Image support is default off. Before enabling it, configure a private image root, PHP upload/request limits and temporary export capacity, then verify backup and restore. Follow Image backup and recovery. Text remains usable on HTTP; image authoring requires HTTPS or localhost Web Crypto. Do not expose the image directory through Apache aliases, symlinks or a public CDN.

Safe Deployment Procedure

1. Deploy the application code first. Do not copy generated HTML into `public/`.
   This version deliberately ignores old sibling `public/*.html` files, so the
   application remains available through its dynamic path before the first new
   release is ready.
2. From the deployed application directory, build and publish the forum
   snapshot:

   ```bash
   time ./v3 build-static
   ```

   The command builds a candidate SQLite model and candidate HTML directory
   outside the live request path, then replaces the database and `current`
   release pointer only after validation. It may take time proportional to the
   repository, but it must not hold the live read-model write lock while it
   walks Git history or renders pages.

   For a presentation-only deployment with an already active release, use the
   fast shared-page refresh instead:

   ```bash
   time ./v3 build-static --shared-only
   ```

   It preserves the existing tag, thread, post, and profile pages, refreshes
   shared pages and their fingerprinted assets, and does not rebuild data. It
   is not a substitute for the complete build after content changes.

   Publish the complete docs library separately on initial setup and after docs,
   renderer, profile or asset changes, with the same destination/profile settings:

   ```bash
   time ./v3 docs publish
   # Explicit destinations (also supports --json):
   ./v3 docs publish --repository-root="$FORUM_REPOSITORY_ROOT" --static-html-root="$FORUM_STATIC_HTML_ROOT"
   ```

   Docs publication changes only the sibling docs release. Failure retains the
   prior library, including a legacy combined-release library; fix the cause and
   retry the same command. A forum build/shared refresh/import never repairs docs.
   Check `/docs/`, a reader/download, and `/api/docs-version`; an open docs page
   should offer Reload after docs publication, without prompting for forum imports.

3. Smoke-test both an anonymous window and an existing signed-in browser:
   `/`, a thread, `/account/key/`, and `/invites/`. A normal reload must be
   sufficient; HTML revalidates with an ETag, while fingerprinted assets are
   immutable and versioned.
4. Confirm the active release rather than editing it:

   ```bash
   readlink "$FORUM_STATIC_HTML_ROOT/current"
   readlink "${FORUM_STATIC_HTML_ROOT}-docs/current"
   ```

If the publish command fails, the previous current release remains active. If a content write occurs during or after a publish, its static pointer is withdrawn and PHP serves current content until the next successful publish. Do not run the old static-build script from a pre-release checkout: upgrade the code before invoking ./v3 build-static.

Deploy from the Site

A root-approved operator can update the application checkout from Tools → Deploy (/tools/deploy/) instead of running git pull over SSH. The same routine runs as ./v3 deploy from the application directory.

• Queue deploy hands the update to the internal task queue. Queued deploys run before any other queued task, so it normally starts at the next cron minute; while it waits, the page says how long it has been queued.

• Deploy now runs it in the web request. Use it for small releases, or when the page says the task worker has not run recently.

• Check for updates fetches from GitHub and shows how far behind origin/main the checkout is. Opening the page never contacts GitHub; it shows the result of the last fetch and when it happened.

After a check finds new commits, the Incoming row shows what a deploy would bring in: the size of the change (files, lines added and removed), and the commits themselves, newest first, up to 20, with merge commits left out. Commits link to the source repository when FORUM_SOURCE_REPOSITORY_URL is a GitHub address. When the update also changes a file the server keeps a local edit to, the row names it, because that is when a deploy can stop with local_changes_conflict. ./v3 deploy --status prints the same as Incoming-* lines.

A deploy is pull only: it fetches origin/main and fast-forwards. It does not run ./v3 docs publish, ./v3 build-static or any other step; run those yourself when the release needs them (see the Safe Deployment Procedure above). With PHP OPcache configured not to check file timestamps, reload PHP after a deploy.

Local edits to tracked files on the server (typically public/.htaccess) are kept: the deploy sets them aside with git stash, fast-forwards, puts them back and drops the stash, the same as running git stash, git pull, git stash pop by hand. The Deploy page lists those files. Untracked files are never touched. If the update changes the same lines as a local edit, the deploy goes back to the previous commit with the edit intact and reports local_changes_conflict; merge that file over SSH.

It refuses, leaving the checkout exactly as it was, when:

• the update conflicts with a local edit (local_changes_conflict, above);

• the checkout is not on main (wrong_branch);

• the server has commits that are not on origin/main (diverged);

• another deploy is running (busy; a queued deploy retries this on a later run).

Fetch failures and timeouts are reported as fetch_failed or fetch_timeout. In each case fix the cause over SSH (or run ./v3 deploy there) and try again. Every attempt, including refusals, is appended to state/logs/app_update.log in Key: value form; ./v3 deploy --status shows the latest.

Requirements: the application directory is a Git checkout whose origin the web and cron users can fetch without a prompt, and the checkout is writable by those users. The feature cannot install itself: the first release that contains it is deployed with an ordinary git pull.

Manual OpenPGP Release Check

Run these checks manually after a release. They are intentionally excluded from CI, cron, browser page loads, and automatic deployment hooks.

1. First run the read-only asset gate from a machine that can reach the public
   site:

   ```bash
   ./v3 openpgp smoke
   ```

   It must report `PASS` for both HTTP (selected OpenPGP v5) and HTTPS
   (selected OpenPGP v6), as well as both legacy raw bundle URLs. A redirect,
   non-JavaScript content type, missing runtime map, or failed request is a
   release failure. Repair and redeploy before continuing; do not compensate
   by clearing browser storage or changing Apache/vhost configuration.

   For a staging investigation, use the staging hostname instead:

   ```bash
   ./v3 openpgp smoke --origin=staging.example
   ```

   Then use separate fresh browser profiles to submit a disposable post through
   both `http://staging.example/compose/thread` (v5) and
   `https://staging.example/compose/thread` (v6). If identity bootstrap fails,
   allow the existing automatic retry to finish and record the browser's
   diagnostic code and attempt ID. Join that ID to the
   `identity_bootstrap_signature_verification_failed` server event; retain only
   its safe attempt, bundle, retry, exit-result, `VALIDSIG`, and GnuPG
   status-code fields. Do not copy key armor, signatures, canonical records,
   raw GnuPG output, or user IDs into an issue. A successful path should create
   one staging identity and post, then reuse that identity without another
   username prompt.

2. Only after the asset gate passes, run the visible new-user canary exactly
   once. From the operator workstation, install the pinned browser driver once
   for the checkout, then pass the local Chromium executable explicitly:

   ```bash
   npm install
   ./v3 openpgp canary --confirm-production-write
   ```

   The command automatically uses `which chromium`. Use any locally installed
   Chromium or Chrome executable; no Snap install is required. Supply
   `--browser-executable=/path/to/chromium` only if the automatic lookup is
   unsuitable. `--headed` is available when a visible browser window is useful.
   The command uses a fresh isolated
   **HTTP** browser profile, creates a release-check identity, publishes
   “New release just dropped, making sure it works,” reloads it, and verifies
   that the same identity is ready without a second username prompt.

   The username is consistently `release-check`; each run still generates a
   fresh browser-held keypair because it uses an isolated browser profile.

   The identity and post are intentional durable community records. If the
   command reports an ambiguous failure after posting, do **not** rerun it:
   inspect its reported post URL and browser output first. HTTP and HTTPS have
   separate browser storage, so this HTTP canary does not substitute for an
   existing HTTPS identity.

Public Session Restoration

Public pages remain readable without a viewer session. A browser that already has an authenticated session resumes it on ordinary HTML navigation. When that session is absent, a browser with a usable saved approved-member key can prove key possession and restore its session automatically; writes and API requests are never retried automatically. Invitation issuance continues to require the restored session plus its existing browser signature verification.

LLM exchange records are private runtime data. Keep LLM_EXCHANGE_DATABASE_PATH outside public/, the canonical repository, and the published read-model database; restrict the file to the deployment/web users. The exchange UI is available only to approved viewers and can be disabled independently with LLM_CONVERSATION_UI_ENABLED=false.

Internal task-queue records are also private runtime data. Keep FORUM_TASK_QUEUE_DATABASE_PATH outside public/, the canonical repository, and the published read-model database. The queue is deliberately separate from the rebuildable read model so a rebuild task retains its state and final outcome. Its executor history, detached-launch outcomes, and rebuild checkpoints live in that same private database; terminal history is bounded while active work and automatic-recovery circuit state are retained.

Visitor statistics are private runtime data. Keep VISITOR_STATISTICS_DATABASE_PATH outside public/, the canonical repository, and published artifacts. It retains only bounded UTC hourly aggregates for up to 90 days: eligible-request counters, anonymous/server-authenticated request counters, and mergeable client/authenticated-user cardinality sketches. It never retains raw visitor, session, key, identity, route, referrer, header, or network records. A deployment begins a new hourly collection epoch; legacy daily aggregates are not converted into hourly history. Its protected page reports only server-observed eligible traffic; direct edge/static delivery outside PHP is not included.

LLM Provider Config

Post analysis and agent reply drafting use provider-neutral LLM_* private config. Create or inspect the private config with:

./v3 private-config --force
./v3 private-config view
./v3 private-config refresh-template

Use refresh-template after upgrades to rewrite the private config with current comments and provider examples while preserving existing effective values.

Private Message Mailbox Operations

PRIVATE_MESSAGE_DATABASE_PATH holds encrypted message envelopes and routing metadata. It is authoritative runtime state: unlike the read model and static artifacts, it cannot be rebuilt from the canonical repository. It must remain outside the document root, canonical repository, static release root, and offline snapshot tree. Encrypted history transfers and sync checkpoints/receipts live separately in PRIVATE_MESSAGE_HISTORY_SYNC_DATABASE_PATH. Preserve both stores: losing retained transfers can remove a newer device's historical access when no donor remains. The sync store initializes lazily and can fail without blocking ordinary messaging. See the history-sync API and operations guide.

Use a dedicated directory readable only by the deployment and web-service users (for example, directory mode 0700 and SQLite database/sidecar mode 0600). Check both databases plus any -journal, -wal and -shm sidecars after deploys or ownership changes. The application does not retain browser private keys; backing up this database preserves envelopes, not users' ability to decrypt a lost browser key.

Quiesce message and sync writes and drain requests across both captures for a common backup point. Resolve both configured paths first. Use SQLite's consistent backup mechanism, not a live raw file copy. For example, with writes stopped:

sqlite3 "$PRIVATE_MESSAGE_DATABASE_PATH" ".backup '/secure/backups/messages-$(date +%F).sqlite3'"
sqlite3 "/secure/backups/messages-$(date +%F).sqlite3" 'PRAGMA integrity_check;'
sqlite3 "$PRIVATE_MESSAGE_HISTORY_SYNC_DATABASE_PATH" ".backup '/secure/backups/message-history-sync-$(date +%F).sqlite3'"
sqlite3 "/secure/backups/message-history-sync-$(date +%F).sqlite3" 'PRAGMA integrity_check;'

Test restoration in a non-production path before relying on a backup. Stop message/sync writes or take the application out of service for the restore, replace both files with their verified matching backups and compatible approval state, restore owner and restrictive permissions, then reopen Messages and verify sent/received history, reload and donor-assisted recovery. Mismatched captures reject changed/orphaned bindings and reopen missing coverage; lost transfers can still require a returning donor. On code rollback retain both files; prior code ignores sync storage and re-upgrade resumes it. Do not place private backups under public/, the canonical repository, or a static release.

There is no automatic message deletion in this release. Set and document a local retention period appropriate to the forum's low-value-content policy; take and validate a backup before a reviewed, targeted retention purge. Keep the retention tooling separate from public artifact publication so purging mailboxes cannot affect posts, profiles, or offline snapshots.

Dedalus Labs, the OpenAI-compatible endpoint previously used as the default for existing installs, has been discontinued. Installs still configured with LLM_PROVIDER => 'dedalus' (or with no LLM_PROVIDER set at all) must pick one of the providers below — there is no default.

Supported providers:

• openai: direct OpenAI API

• openrouter: OpenRouter API, with optional attribution headers

• anthropic: direct Anthropic Messages API

• stub: deterministic local analysis for smoke tests and offline operation

• custom OpenAI-compatible gateways such as LiteLLM, vLLM, llama.cpp, or internal routers

The common keys are:

'LLM_PROVIDER' => '', // required: openai, openrouter, anthropic, stub, or a custom gateway name
'LLM_API_KEY' => 'replace-with-real-key',
'LLM_API_BASE_URL' => '', // required for every provider except stub; see examples below
'LLM_MODEL' => '', // required for every provider except stub; see examples below
'LLM_TIMEOUT_SECONDS' => 60,
'LLM_EXTRA_HEADERS' => [],
'LLM_POST_ANALYSIS_PROMPT_PATH' => 'prompts/dedalus_post_analysis_system.txt',

Provider examples:

// Direct OpenAI
'LLM_PROVIDER' => 'openai',
'LLM_API_BASE_URL' => 'https://api.openai.com',
'LLM_MODEL' => 'gpt-5-nano',

// Direct Anthropic
'LLM_PROVIDER' => 'anthropic',
'LLM_API_BASE_URL' => 'https://api.anthropic.com',
'LLM_MODEL' => 'claude-haiku-4-5-20251001',

// OpenRouter
'LLM_PROVIDER' => 'openrouter',
'LLM_API_BASE_URL' => 'https://openrouter.ai/api',
'LLM_MODEL' => 'openai/gpt-5-nano',
'LLM_EXTRA_HEADERS' => [
    'HTTP-Referer' => 'https://forum.example',
    'X-Title' => 'Forum',
],

// LiteLLM or another OpenAI-compatible gateway
'LLM_PROVIDER' => 'litellm',
'LLM_API_BASE_URL' => 'https://llm-gateway.example',
'LLM_MODEL' => 'openai/gpt-5-nano',

OpenAI-compatible providers are called at LLM_API_BASE_URL + /v1/chat/completions. Anthropic is called at LLM_API_BASE_URL + /v1/messages. Legacy DEDALUS_* LLM settings are still read as fallbacks for current deployments, but new config writes use LLM_*, and LLM_PROVIDER must now be set explicitly since Dedalus Labs is no longer available.

Agent Reply Requests

Approved users can request a reply-agent response from eligible post cards. The button records a durable request and one general-purpose agent_reply task, then returns without waiting for provider analysis or canonical posting.

The task-queue worker fulfills the task with the same repository, database, private LLM config, reply-agent key directory, and artifact paths as the web app. Its exclusive lock, per-reply task identity, and reply publication reservation prevent duplicate posts. A worker run also reconciles durable requested rows that lack an active task.

Useful manual commands:

./v3 agent-reply test
./v3 agent-reply test-local
./v3 agent-reply status <post-id>
./v3 agent-reply status --limit=25
./v3 task-queue status
./v3 task-queue run --dry-run
./v3 task-queue run --limit=1

Internal Task Queue

The internal task queue handles allowlisted maintenance work outside visitor requests, including requested agent replies, read-model rebuild/recovery, Fastmod sweeps, public offline-snapshot publication and operator-requested deploys, which it claims ahead of other queued work. It does not run user-supplied commands and remains separate from the Codex-handoff queue.

Install the cron worker with the current-path reference:

./v3 task-queue cron

Remove any legacy agent-reply cron entry, then install only the line printed by this command. The worker processes one task per minute and exits successfully when another queue worker is active. The default cron-output path is application-private (state/private/task_queue_cron.log), so ensure its directory is writable by the cron user before installing the line. Successful site updates enqueue the deduplicated offline-snapshot task. Useful operator commands are:

./v3 task-queue enqueue-rebuild
./v3 task-queue enqueue-offline-snapshot
./v3 task-queue reset-recovery
./v3 task-queue status
./v3 task-queue run --dry-run
./v3 task-queue run --limit=1

Site Feature Flags

The site exposes registered public flags at /tools/feature-flags/.

Root-approved users can change mutable site flags from that page. Successful changes update:

records/instance/feature-flags.txt

and commit the change to the content repository git log.

Runtime precedence is:

1. environment/private override

2. git-backed site value

3. code default

Use FORUM_* environment variables for emergency or deployment-level overrides. While an environment variable is present, the corresponding flag is effectively pinned by the process and the site UI reports the environment source.

Approved-members-only access

FORUM_APPROVED_MEMBERS_ONLY=true enables the private-site boundary independently of the site profile or theme. Unapproved visitors are limited to /lobby/, /account/key/, and their own authenticated /profiles/<slug> page. When a browser keypair is already saved, Lobby automatically publishes the public key and completes identity setup before authentication. Other routes, feeds, APIs, downloads, backups, and generated HTML return 404 or are routed through PHP for the access decision. Required static assets remain directly servable.

Enable it in the instance feature-flags record:

FORUM_APPROVED_MEMBERS_ONLY: true

For a deployment-level pin, set FORUM_APPROVED_MEMBERS_ONLY=true in the vhost environment. Keep the flag off for public instances. The checked-in rewrite rules route every non-asset request through PHP regardless of whether the effective flag comes from the vhost or the instance feature-flags record, so old public HTML artifacts cannot bypass a site-level flag change.

Audit site-level changes with:

git -C /srv/forum-rewrite/repository log -- records/instance/feature-flags.txt
git -C /srv/forum-rewrite/repository show <commit>:records/instance/feature-flags.txt

Rollback options:

• set the previous value through /tools/feature-flags/

• or revert the relevant content-repository commit

If production serves prebuilt static HTML artifacts, rebuild artifacts after changing flags outside the web write path. Private instances do not serve those content artifacts; every content request reaches the application access gate first.

Automatic guest keypairs

FORUM_AUTOMATIC_GUEST_KEYPAIR_ENABLED defaults to false. When enabled through /tools/feature-flags/, the instance feature-flags record, or a deployment environment override, a fresh browser receives a browser-local guest keypair before its first signed action. The server receives only public-key publication requests; it never receives the private key.

The browser-level Account Key setting controls publication timing. Its default is first signed use, which avoids creating public identities for visitors who never interact. A visitor may explicitly switch it to immediate publication. Existing browser keypairs are retained and are not replaced by this feature.

For a deployment-level enablement:

FORUM_AUTOMATIC_GUEST_KEYPAIR_ENABLED=true ./v3 start

To roll back, disable the flag on /tools/feature-flags/ or remove/set the environment override to false; this stops new automatic generation but intentionally does not delete keypairs already stored in visitors' browsers. Rebuild static artifacts after changing the site-record value outside the web write path.

Redeemable board invitations

Approved, authenticated members can use Invite from any board page to generate a signed, single-use invitation. New links use a 128-bit, 32-character hexadecimal bearer secret in the browser fragment; share it only with the intended recipient. The Activity feed records issued, revoked, and redeemed events with the verification hash, never the secret or the share URL.

Recipients create a disposable browser key in Lobby, redeem the invitation, authenticate with that key, and then enter the selected internal destination or the board. Invitations expire after seven days. An issuer can revoke an unused invitation from the Invite page using its invitation ID and verification hash.

If an invitation is sent to the wrong person or its secret leaks, revoke it immediately. A redeemed invitation has already produced an auditable approval; use the normal membership/approval recovery process rather than deleting canonical invitation records. Never paste bearer secrets into board posts, Activity, tickets, logs, or shell history.

App Version Notification

FORUM_APP_VERSION_NOTIFICATION=false disables the browser-side /api/version polling and the "A new version is available." reload banner.

The default is enabled. /api/version remains available when the notification is disabled.

If production serves prebuilt static HTML artifacts, rebuild those artifacts after changing this flag so rendered pages include or omit the notification markup consistently.

HTTP, HTTPS, and Browser OpenPGP

Plain HTTP remains a supported production access path. The default Apache example serves the app on port 80 and must not be replaced with a forced HTTP-to-HTTPS redirect unless the local operator intentionally accepts the extra TLS failure modes.

Do not enable HSTS by default. Strict-Transport-Security can prevent users from reaching the site after certificate expiry, hostname changes, proxy mistakes, subdomain gaps, captive-portal interference, or other TLS failures.

Browser-held OpenPGP identity uses:

• public/assets/openpgp_loader.js

• public/assets/openpgp.min.js for OpenPGP.js v6 on HTTPS and secure loopback origins

• public/assets/openpgp.v5.11.3.min.js as the patched v5 fallback on public HTTP

OpenPGP.js v6 is preferred, but browsers do not expose crypto.subtle on public HTTP origins. The loader therefore uses the v5 fallback on public HTTP so authored browser-identity posting can still work where the legacy bundle works.

If browser OpenPGP is unavailable, compose forms expose an explicit anonymous submit button. That path submits with an empty author_identity_id; the server writes a normal canonical post without Author-Identity-ID. This is intentional and is separate from authored OpenPGP posting.

Server-side user identity generation is intentionally out of scope. Do not generate, store, or manage user private keys on the server for this fallback.

Unicode Authored Text Rollout

FORUM_UNICODE_AUTHORED_TEXT=true enables visible UTF-8 prose in post subjects and bodies.

Scope:

• affected: human-authored post subject and body

• unchanged: post IDs, thread IDs, parent IDs, board tags, reaction tags, profile slugs, identity IDs, routes, and artifact paths

The server still rejects invalid UTF-8, control characters, format characters, bidirectional controls, private-use characters, noncharacters, unsupported spacing characters, and symbols such as emoji.

Operational notes:

• keep the flag disabled for first deploys unless Unicode prose has been explicitly tested on the target host

• verify git diff, rebuild scripts, static artifact generation, RSS, and terminal inspection all run under a UTF-8-capable locale

• PHP intl is optional in this implementation; when it is unavailable, combining-mark input is rejected instead of normalized, while precomposed readable Unicode such as normal Cyrillic remains supported

• rollback is FORUM_UNICODE_AUTHORED_TEXT=false; existing Unicode records remain valid UTF-8 canonical text and should continue to parse and render

Rollout smoke test:

1. enable FORUM_UNICODE_AUTHORED_TEXT=true

2. create a test thread with subject Привет and body Привет мир

3. verify the thread page, post page, activity page, RSS feed, read-model rebuild, and static artifact build

4. verify a post with a zero-width or bidirectional control character is rejected

5. disable the flag if submit-time Unicode acceptance must be rolled back

Automatic Agent Replies

Automatic replies can be disabled with:

DEDALUS_AGENT_REPLIES_ENABLED=false

Agent replies are drafted by the post-analysis prompt and posted from the stored engagement.suggested_response when respondability gates pass. Production uses the configured LLM_* provider for that analysis call.

On first successful agent reply, the app bootstraps a canonical reply-agent OpenPGP identity and stores the private key under:

<application-root>/state/private/agent-reply/

This directory must not be under public/ and should be readable only by the deployment user and web user. To rotate the key, disable automatic replies, archive the old private key, remove or supersede the canonical reply-agent identity through an operator-reviewed migration, rebuild the read model, then re-enable replies so a new key can be bootstrapped.

First-Time Setup

1. Check out the application code.

2. Create the writable canonical repository checkout.

3. Ensure the repository is a real git checkout with records/ and .git/.

4. Create the writable state directory.

5. Configure Apache to serve public/.

6. Put the deployment settings in the checkout's private config file and check them with ./v3 status.

7. Run the initial read-model rebuild. On a public instance, it automatically publishes the first offline snapshot when one is not already served; treat a reported snapshot-readiness failure as a launch blocker.

8. Optionally build sibling static HTML artifacts.

Example commands:

php scripts/rebuild_read_model.php /srv/forum-rewrite/repository /srv/forum-rewrite/state/cache/post_index.sqlite3
php scripts/build_static_artifacts.php /srv/forum-rewrite/repository /srv/forum-rewrite/state/cache/post_index.sqlite3 /srv/forum-rewrite/app/public

For a public instance, before launch verify the anonymous endpoint from an operator machine:

./v3 offline diagnose --url=https://your-public-domain

The command must report a successful public snapshot response. In approved-members-only mode, public snapshots are intentionally unavailable.

Pre-Launch Checklist

• Apache DocumentRoot points to public/

• AllowOverride permits .htaccess if using the checked-in rewrite file

• mod_rewrite is enabled

• PHP can open PDO SQLite

• the writable repository is a git checkout

• the web user can create commits in the writable repository

• the web user can write the read-model database and lock files

• the web user can invalidate public/*.html artifacts if sibling artifacts are enabled

• /api/read_model_status returns status=ready

• on public instances, ./v3 offline diagnose --url=https://your-public-domain reports a successful public snapshot response

• HTTP requests to /account/key/, /compose/thread, and /assets/openpgp_loader.js return app/asset responses, not forced HTTPS redirects

• default responses do not emit Strict-Transport-Security

Manual Verification

Before launch, verify:

• board route loads

• thread route loads

• anonymous /offline/snapshot.sqlite3 returns a successful response on public instances

• profile route loads

• account route loads

• compose thread/reply routes load

• anonymous queryless board/thread/profile requests can be served from sibling *.html artifacts through the front controller

• cookie-bearing requests bypass static artifacts and fall back to PHP

• thread creation works

• reply creation works

• authored browser identity bootstrap works on HTTPS or secure loopback with the v6 OpenPGP path

• authored browser identity bootstrap works on public HTTP with the v5 OpenPGP fallback where supported by the browser

• explicit anonymous compose works when browser OpenPGP is unavailable

Recommended Launch Sequence

1. deploy application code

2. verify Apache config

3. verify writable repository/config paths

4. rebuild read model and resolve any offline snapshot-readiness failure

5. build static artifacts

6. open /api/read_model_status

7. smoke-test core read routes

8. smoke-test one write flow

Related Docs

• operator_recovery.md

• apache_vhost.conf

• env.production.example

On-demand content import

After deployment, operators can use ./v3 import-instance to download and merge public forum content from another instance. No migration or cron setup is needed. Use the running instance's repository/database/static-root configuration and validate a disposable import with the target host's PHP, SQLite, zlib/OpenSSL, Git, and gzip tooling first. Follow the import deployment check for preview, coverage review, served-view verification, and repeat-run checks.

Retraction sweep operations

See local cleanup coverage and recovery. Deploy web/worker changes together and keep the existing single queue cron entry. Preview then run immediately or enqueue a sweep; inspect status and retry failures without signing another retraction. Canonical/history and backup removal are outside this operation. Pause/drain writers and workers before rollback; preserve signed evidence and avoid restoring diagnostic payloads.

Frontend status target

After deployment, run ./v3 status from the application checkout with its normal repository/database environment. A unique verified loopback port-80 endpoint is detected automatically. If the site requires a virtual host or HTTPS, set FORUM_STATUS_URL in the existing private configuration to its full URL, then rerun plain status and confirm the URL/source and Frontend verdict. Use --url=... for a one-time override. Discovery WARN means the intended endpoint could not be established; a configured endpoint's outage remains FAIL. See operator recovery.