zenmemes

Platform Docs

Repository source: docs/runbooks/production_deploy.md

Production Deploy Runbook

This runbook describes the intended production deployment model for the PHP forum rewrite.

Deployment Model

The intended production shape is:

  • Apache serves public/ as the DocumentRoot
  • PHP handles dynamic requests through public/index.php
  • Apache directly serves only existing /assets/* files and favicon.ico
  • the PHP front controller serves eligible pages from an atomically selected static release
  • the canonical writable repository lives outside public/
  • derived state under state/ is writable by the web user

This application assumes a conservative shared-host style environment with:

  • PHP 8.1+
  • PDO SQLite
  • standard filesystem functions
  • shell access sufficient for non-interactive git commands
  • Apache with mod_rewrite

Required Directory Layout

One workable layout:

/srv/forum-rewrite/
  app/                      application checkout
    public/
    src/
    templates/
    scripts/
  repository/               writable canonical content checkout
    records/
    .git/
  state/
    cache/
    private/
    static_html/
      releases/
      current -> releases/release-...   active complete HTML release
    forum-rewrite.lock
    read_model_stale.json

Recommended mapping:

  • application root: /srv/forum-rewrite/app
  • Apache DocumentRoot: /srv/forum-rewrite/app/public
  • writable repository root: /srv/forum-rewrite/repository
  • read-model database: /srv/forum-rewrite/state/cache/post_index.sqlite3
  • static release root: /srv/forum-rewrite/state/static_html

Optional runtime setting:

  • FORUM_EXECUTION_LOCK_TIMEOUT_SECONDS: seconds a request waits for the shared write/read-model lock before returning a busy error. The default is 5.
  • FORUM_UNICODE_AUTHORED_TEXT: when set to true, subject/body prose may contain visible UTF-8 text such as Cyrillic. The default is disabled.
  • FORUM_APP_VERSION_NOTIFICATION: when set to false, disables browser-side app version polling and the reload banner. The default is enabled.
  • LLM_CONVERSATION_RECORDING_ENABLED: controls private exact-prompt/response capture. The default is enabled.
  • LLM_CONVERSATION_UI_ENABLED: controls approved-user/operator web visibility of captured exchanges. The default is enabled.
  • LLM_EXCHANGE_DATABASE_PATH: optional private SQLite path; defaults to <application-root>/state/private/llm_exchanges.sqlite3.
  • FORUM_TASK_QUEUE_DATABASE_PATH: optional private SQLite path for queued internal maintenance; defaults to <application-root>/state/private/internal_tasks.sqlite3.
  • FAST_SCORING_DATABASE_PATH: optional private SQLite score-state path; defaults to <application-root>/state/private/fast_scores.sqlite3.
  • FAST_SCORING_AUTOMATIC_ENQUEUE_ENABLED: when true alongside FAST_SCORING_ENABLED, creates private score work for newly published posts only; defaults to false.

Writable Paths

The web user must be able to write:

  • the canonical repository checkout at FORUM_REPOSITORY_ROOT
  • the parent directory of FORUM_DATABASE_PATH
  • the lock file directory next to FORUM_DATABASE_PATH
  • FORUM_STATIC_HTML_ROOT and its releases/ directory
  • state/private/agent-reply/ under the application root if agent reply fulfillment is enabled
  • the parent directory of LLM_EXCHANGE_DATABASE_PATH if LLM conversation recording is enabled
  • the parent directory of FORUM_TASK_QUEUE_DATABASE_PATH when the internal task queue is enabled
  • the parent directory of FAST_SCORING_DATABASE_PATH when Fastmod sweeps are enabled

Static HTML is derived state. A write removes the current release pointer, so subsequent public requests use PHP until a fresh complete release is published. Old release directories are retained and are never edited in place.

Environment Variables

Use these in production:

  • FORUM_REPOSITORY_ROOT
  • FORUM_DATABASE_PATH
  • FORUM_STATIC_HTML_ROOT

Suggested values:

FORUM_REPOSITORY_ROOT=/srv/forum-rewrite/repository
FORUM_DATABASE_PATH=/srv/forum-rewrite/state/cache/post_index.sqlite3
FORUM_STATIC_HTML_ROOT=/srv/forum-rewrite/state/static_html

FORUM_STATIC_HTML_ROOT defaults to <application-root>/state/static_html. It is not under public/; Apache continues to route content requests through PHP, which selects only the current release.

Safe Deployment Procedure

  1. Deploy the application code first. Do not copy generated HTML into public/.

This version deliberately ignores old sibling public/*.html files, so the application remains available through its dynamic path before the first new release is ready.

  1. From the deployed application directory, build and publish the complete

snapshot:

   time ./v3 build-static

The command builds a candidate SQLite model and candidate HTML directory outside the live request path, then replaces the database and current release pointer only after validation. It may take time proportional to the repository, but it must not hold the live read-model write lock while it walks Git history or renders pages.

For a presentation-only deployment with an already active release, use the fast shared-page refresh instead:

   time ./v3 build-static --shared-only

It preserves the existing tag, thread, post, and profile pages, refreshes shared pages and their fingerprinted assets, and does not rebuild data. It is not a substitute for the complete build after content changes.

  1. Smoke-test both an anonymous window and an existing signed-in browser:

/, a thread, /account/key/, and /invites/. A normal reload must be sufficient; HTML revalidates with an ETag, while fingerprinted assets are immutable and versioned.

  1. Confirm the active release rather than editing it:
   readlink "$FORUM_STATIC_HTML_ROOT/current"

If the publish command fails, the previous current release remains active. If a content write occurs during or after a publish, its static pointer is withdrawn and PHP serves current content until the next successful publish. Do not run the old static-build script from a pre-release checkout: upgrade the code before invoking ./v3 build-static.

Manual OpenPGP Release Check

Run these checks manually after a release. They are intentionally excluded from CI, cron, browser page loads, and automatic deployment hooks.

  1. First run the read-only asset gate from a machine that can reach the public

site:

   ./v3 openpgp smoke

It must report PASS for both HTTP (selected OpenPGP v5) and HTTPS (selected OpenPGP v6), as well as both legacy raw bundle URLs. A redirect, non-JavaScript content type, missing runtime map, or failed request is a release failure. Repair and redeploy before continuing; do not compensate by clearing browser storage or changing Apache/vhost configuration.

  1. Only after the asset gate passes, run the visible new-user canary exactly

once. From the operator workstation, install the pinned browser driver once for the checkout, then pass the local Chromium executable explicitly:

   npm install
   ./v3 openpgp canary --confirm-production-write

The command automatically uses which chromium. Use any locally installed Chromium or Chrome executable; no Snap install is required. Supply --browser-executable=/path/to/chromium only if the automatic lookup is unsuitable. --headed is available when a visible browser window is useful. The command uses a fresh isolated HTTP browser profile, creates a release-check identity, publishes “New release just dropped, making sure it works,” reloads it, and verifies that the same identity is ready without a second username prompt.

The username is consistently release-check; each run still generates a fresh browser-held keypair because it uses an isolated browser profile.

The identity and post are intentional durable community records. If the command reports an ambiguous failure after posting, do not rerun it: inspect its reported post URL and browser output first. HTTP and HTTPS have separate browser storage, so this HTTP canary does not substitute for an existing HTTPS identity.

Public Session Restoration

Public pages remain readable without a viewer session. A browser that already has an authenticated session resumes it on ordinary HTML navigation. When that session is absent, a browser with a usable saved approved-member key can prove key possession and restore its session automatically; writes and API requests are never retried automatically. Invitation issuance continues to require the restored session plus its existing browser signature verification.

LLM exchange records are private runtime data. Keep LLM_EXCHANGE_DATABASE_PATH outside public/, the canonical repository, and the published read-model database; restrict the file to the deployment/web users. The exchange UI is available only to approved viewers and can be disabled independently with LLM_CONVERSATION_UI_ENABLED=false.

Internal task-queue records are also private runtime data. Keep FORUM_TASK_QUEUE_DATABASE_PATH outside public/, the canonical repository, and the published read-model database. The queue is deliberately separate from the rebuildable read model so a rebuild task retains its state and final outcome.

Site Profile

FORUM_SITE_ID selects which SiteProfileRegistry entry (site name, default theme, composer copy) this deployment renders. Values: zenmemes, chouse. Unset, empty, or unrecognized values resolve to zenmemes.

Set it alongside the three path variables above, per vhost:

FORUM_SITE_ID=zenmemes

It only selects branding — it does not select content or database paths. Those remain on FORUM_REPOSITORY_ROOT, FORUM_DATABASE_PATH, and FORUM_STATIC_HTML_ROOT, set independently per vhost. A vhost with a mismatched FORUM_SITE_ID and content paths is a misconfiguration, not a supported mode; the two must be kept paired by whoever edits the vhost config.

Precedence and rollback match the other FORUM_* variables in this runbook: environment variable wins, code default (zenmemes) applies when absent, and rollback is deleting the SetEnv line.

For local/CLI runs, set the same variable before the command:

FORUM_SITE_ID=chouse ./v3 start

Omitting FORUM_REPOSITORY_ROOT and FORUM_DATABASE_PATH uses the same default local instance state for every site profile. Only the disposable static HTML cache remains profile-specific by default, preventing rendered presentation from crossing profiles. Use explicit repository and database paths when developing genuinely separate instances from one checkout.

LLM Provider Config

Post analysis and agent reply drafting use provider-neutral LLM_* private config. Create or inspect the private config with:

./v3 private-config --force
./v3 private-config view
./v3 private-config refresh-template

Use refresh-template after upgrades to rewrite the private config with current comments and provider examples while preserving existing effective values.

Dedalus Labs, the OpenAI-compatible endpoint previously used as the default for existing installs, has been discontinued. Installs still configured with LLM_PROVIDER => 'dedalus' (or with no LLM_PROVIDER set at all) must pick one of the providers below — there is no default.

Supported providers:

  • openai: direct OpenAI API
  • openrouter: OpenRouter API, with optional attribution headers
  • anthropic: direct Anthropic Messages API
  • stub: deterministic local analysis for smoke tests and offline operation
  • custom OpenAI-compatible gateways such as LiteLLM, vLLM, llama.cpp, or internal routers

The common keys are:

'LLM_PROVIDER' => '', // required: openai, openrouter, anthropic, stub, or a custom gateway name
'LLM_API_KEY' => 'replace-with-real-key',
'LLM_API_BASE_URL' => '', // required for every provider except stub; see examples below
'LLM_MODEL' => '', // required for every provider except stub; see examples below
'LLM_TIMEOUT_SECONDS' => 60,
'LLM_EXTRA_HEADERS' => [],
'LLM_POST_ANALYSIS_PROMPT_PATH' => 'prompts/dedalus_post_analysis_system.txt',

Provider examples:

// Direct OpenAI
'LLM_PROVIDER' => 'openai',
'LLM_API_BASE_URL' => 'https://api.openai.com',
'LLM_MODEL' => 'gpt-5-nano',

// Direct Anthropic
'LLM_PROVIDER' => 'anthropic',
'LLM_API_BASE_URL' => 'https://api.anthropic.com',
'LLM_MODEL' => 'claude-haiku-4-5-20251001',

// OpenRouter
'LLM_PROVIDER' => 'openrouter',
'LLM_API_BASE_URL' => 'https://openrouter.ai/api',
'LLM_MODEL' => 'openai/gpt-5-nano',
'LLM_EXTRA_HEADERS' => [
    'HTTP-Referer' => 'https://forum.example',
    'X-Title' => 'Forum',
],

// LiteLLM or another OpenAI-compatible gateway
'LLM_PROVIDER' => 'litellm',
'LLM_API_BASE_URL' => 'https://llm-gateway.example',
'LLM_MODEL' => 'openai/gpt-5-nano',

OpenAI-compatible providers are called at LLM_API_BASE_URL + /v1/chat/completions. Anthropic is called at LLM_API_BASE_URL + /v1/messages. Legacy DEDALUS_* LLM settings are still read as fallbacks for current deployments, but new config writes use LLM_*, and LLM_PROVIDER must now be set explicitly since Dedalus Labs is no longer available.

Agent Reply Requests

Approved users can request a reply-agent response from eligible post cards. The button records a durable request in SQLite and returns quickly; it does not wait for provider analysis or canonical posting.

Fulfillment is handled by a periodic PHP command. A typical cron entry is:

* * * * * cd /srv/forum-rewrite/app && php scripts/run_agent_reply_requests.php --quiet --limit=10 >> /var/log/forum-agent-replies.log 2>&1

The deployed checkout can also print the current-path reference:

./v3 agent-reply cron

The command uses the same repository, database, private LLM config, reply-agent key directory, and artifact paths as the web app. It exits successfully if another fulfillment run is already active, and queued-row claims prevent duplicate reply-agent posts for the same requested post content.

Without --quiet, the command prints repository/database/artifact paths, queue counts before and after the run, claimed-row count, one processing/result line per request, reason totals, and elapsed time. --quiet suppresses successful STDOUT for cron while preserving STDERR errors.

Useful manual commands:

./v3 agent-reply test
./v3 agent-reply test-local
php scripts/run_agent_reply_requests.php --dry-run
php scripts/run_agent_reply_requests.php --limit=10
php scripts/run_agent_reply_requests.php --post-id=<post-id>
./v3 agent-reply status <post-id>
./v3 agent-reply status --limit=25
./v3 agent-reply cron run --limit=10
./v3 agent-reply cron run --dry-run

Internal Task Queue

The internal task queue handles allowlisted maintenance work outside visitor requests, including read-model rebuild/recovery, Fastmod sweeps, and public offline-snapshot publication. It does not run user-supplied commands and is separate from the agent-reply and Codex-handoff queues.

Install the cron worker with the current-path reference:

./v3 task-queue cron

Install the line printed by the command. The worker processes one task per minute and exits successfully when another queue worker is active. Successful site updates enqueue the deduplicated offline-snapshot task. Useful operator commands are:

./v3 task-queue enqueue-rebuild
./v3 task-queue enqueue-offline-snapshot
./v3 task-queue status
./v3 task-queue run --dry-run
./v3 task-queue run --limit=1

Site Feature Flags

The site exposes registered public flags at /tools/feature-flags/.

Root-approved users can change mutable site flags from that page. Successful changes update:

records/instance/feature-flags.txt

and commit the change to the content repository git log.

Runtime precedence is:

  1. environment/private override
  2. git-backed site value
  3. code default

Use FORUM_* environment variables for emergency or deployment-level overrides. While an environment variable is present, the corresponding flag is effectively pinned by the process and the site UI reports the environment source.

Approved-members-only access

FORUM_APPROVED_MEMBERS_ONLY=true enables the private-site boundary independently of the site profile or theme. Unapproved visitors are limited to /lobby/, /account/key/, and their own authenticated /profiles/<slug> page. When a browser keypair is already saved, Lobby automatically publishes the public key and completes identity setup before authentication. Other routes, feeds, APIs, downloads, backups, and generated HTML return 404 or are routed through PHP for the access decision. Required static assets remain directly servable.

Enable it in the instance feature-flags record:

FORUM_APPROVED_MEMBERS_ONLY: true

For a deployment-level pin, set FORUM_APPROVED_MEMBERS_ONLY=true in the vhost environment. Keep the flag off for public instances. The checked-in rewrite rules route every non-asset request through PHP regardless of whether the effective flag comes from the vhost or the instance feature-flags record, so old public HTML artifacts cannot bypass a site-level flag change.

Audit site-level changes with:

git -C /srv/forum-rewrite/repository log -- records/instance/feature-flags.txt
git -C /srv/forum-rewrite/repository show <commit>:records/instance/feature-flags.txt

Rollback options:

  • set the previous value through /tools/feature-flags/
  • or revert the relevant content-repository commit

If production serves prebuilt static HTML artifacts, rebuild artifacts after changing flags outside the web write path. Private instances do not serve those content artifacts; every content request reaches the application access gate first.

Redeemable board invitations

Approved, authenticated members can use Invite from any board page to generate a signed, single-use invitation. New links use a 128-bit, 32-character hexadecimal bearer secret in the browser fragment; share it only with the intended recipient. The Activity feed records issued, revoked, and redeemed events with the verification hash, never the secret or the share URL.

Recipients create a disposable browser key in Lobby, redeem the invitation, authenticate with that key, and then enter the selected internal destination or the board. Invitations expire after seven days. An issuer can revoke an unused invitation from the Invite page using its invitation ID and verification hash.

If an invitation is sent to the wrong person or its secret leaks, revoke it immediately. A redeemed invitation has already produced an auditable approval; use the normal membership/approval recovery process rather than deleting canonical invitation records. Never paste bearer secrets into board posts, Activity, tickets, logs, or shell history.

App Version Notification

FORUM_APP_VERSION_NOTIFICATION=false disables the browser-side /api/version polling and the "A new version is available." reload banner.

The default is enabled. /api/version remains available when the notification is disabled.

If production serves prebuilt static HTML artifacts, rebuild those artifacts after changing this flag so rendered pages include or omit the notification markup consistently.

HTTP, HTTPS, and Browser OpenPGP

Plain HTTP remains a supported production access path. The default Apache example serves the app on port 80 and must not be replaced with a forced HTTP-to-HTTPS redirect unless the local operator intentionally accepts the extra TLS failure modes.

Do not enable HSTS by default. Strict-Transport-Security can prevent users from reaching the site after certificate expiry, hostname changes, proxy mistakes, subdomain gaps, captive-portal interference, or other TLS failures.

Browser-held OpenPGP identity uses:

  • public/assets/openpgp_loader.js
  • public/assets/openpgp.min.js for OpenPGP.js v6 on HTTPS and secure loopback origins
  • public/assets/openpgp.v5.11.3.min.js as the patched v5 fallback on public HTTP

OpenPGP.js v6 is preferred, but browsers do not expose crypto.subtle on public HTTP origins. The loader therefore uses the v5 fallback on public HTTP so authored browser-identity posting can still work where the legacy bundle works.

If browser OpenPGP is unavailable, compose forms expose an explicit anonymous submit button. That path submits with an empty author_identity_id; the server writes a normal canonical post without Author-Identity-ID. This is intentional and is separate from authored OpenPGP posting.

Server-side user identity generation is intentionally out of scope. Do not generate, store, or manage user private keys on the server for this fallback.

Unicode Authored Text Rollout

FORUM_UNICODE_AUTHORED_TEXT=true enables visible UTF-8 prose in post subjects and bodies.

Scope:

  • affected: human-authored post subject and body
  • unchanged: post IDs, thread IDs, parent IDs, board tags, reaction tags, profile slugs, identity IDs, routes, and artifact paths

The server still rejects invalid UTF-8, control characters, format characters, bidirectional controls, private-use characters, noncharacters, unsupported spacing characters, and symbols such as emoji.

Operational notes:

  • keep the flag disabled for first deploys unless Unicode prose has been explicitly tested on the target host
  • verify git diff, rebuild scripts, static artifact generation, RSS, and terminal inspection all run under a UTF-8-capable locale
  • PHP intl is optional in this implementation; when it is unavailable, combining-mark input is rejected instead of normalized, while precomposed readable Unicode such as normal Cyrillic remains supported
  • rollback is FORUM_UNICODE_AUTHORED_TEXT=false; existing Unicode records remain valid UTF-8 canonical text and should continue to parse and render

Rollout smoke test:

  1. enable FORUM_UNICODE_AUTHORED_TEXT=true
  2. create a test thread with subject Привет and body Привет мир
  3. verify the thread page, post page, activity page, RSS feed, read-model rebuild, and static artifact build
  4. verify a post with a zero-width or bidirectional control character is rejected
  5. disable the flag if submit-time Unicode acceptance must be rolled back

Automatic Agent Replies

Automatic replies can be disabled with:

DEDALUS_AGENT_REPLIES_ENABLED=false

Agent replies are drafted by the post-analysis prompt and posted from the stored engagement.suggested_response when respondability gates pass. Production uses the configured LLM_* provider for that analysis call.

On first successful agent reply, the app bootstraps a canonical reply-agent OpenPGP identity and stores the private key under:

<application-root>/state/private/agent-reply/

This directory must not be under public/ and should be readable only by the deployment user and web user. To rotate the key, disable automatic replies, archive the old private key, remove or supersede the canonical reply-agent identity through an operator-reviewed migration, rebuild the read model, then re-enable replies so a new key can be bootstrapped.

First-Time Setup

  1. Check out the application code.
  2. Create the writable canonical repository checkout.
  3. Ensure the repository is a real git checkout with records/ and .git/.
  4. Create the writable state directory.
  5. Configure Apache to serve public/.
  6. Set the production environment variables.
  7. Run the initial read-model rebuild.
  8. Optionally build sibling static HTML artifacts.

Example commands:

php scripts/rebuild_read_model.php /srv/forum-rewrite/repository /srv/forum-rewrite/state/cache/post_index.sqlite3
php scripts/build_static_artifacts.php /srv/forum-rewrite/repository /srv/forum-rewrite/state/cache/post_index.sqlite3 /srv/forum-rewrite/app/public

Pre-Launch Checklist

  • Apache DocumentRoot points to public/
  • AllowOverride permits .htaccess if using the checked-in rewrite file
  • mod_rewrite is enabled
  • PHP can open PDO SQLite
  • the writable repository is a git checkout
  • the web user can create commits in the writable repository
  • the web user can write the read-model database and lock files
  • the web user can invalidate public/*.html artifacts if sibling artifacts are enabled
  • /api/read_model_status returns status=ready
  • HTTP requests to /account/key/, /compose/thread, and /assets/openpgp_loader.js return app/asset responses, not forced HTTPS redirects
  • default responses do not emit Strict-Transport-Security

Manual Verification

Before launch, verify:

  • board route loads
  • thread route loads
  • profile route loads
  • account route loads
  • compose thread/reply routes load
  • anonymous queryless board/thread/profile requests can be served from sibling *.html artifacts through the front controller
  • cookie-bearing requests bypass static artifacts and fall back to PHP
  • thread creation works
  • reply creation works
  • authored browser identity bootstrap works on HTTPS or secure loopback with the v6 OpenPGP path
  • authored browser identity bootstrap works on public HTTP with the v5 OpenPGP fallback where supported by the browser
  • explicit anonymous compose works when browser OpenPGP is unavailable

Recommended Launch Sequence

  1. deploy application code
  2. verify Apache config
  3. verify writable repository/config paths
  4. rebuild read model
  5. build static artifacts
  6. open /api/read_model_status
  7. smoke-test core read routes
  8. smoke-test one write flow

Related Docs