# First-server bootstrap runbook

- Status: Approved procedure specification; commands/artifacts not yet implemented
- Target: `zarinpal.amiirkhl.ir`
- Owner: Platform operator
- Last updated: 2026-08-20

This is the safe order for preparing a new production host. It is intentionally a human-reviewed checklist, not a copy-paste bootstrap script. Stop on any failed validation and preserve the existing server/DNS state until the failure is understood.

## Preconditions and recorded inputs

Before connecting to the host, record in the change ticket/release record:

- server provider, instance id, public IPv4/IPv6, region, CPU, RAM, disk type/size, and console/recovery access
- Ubuntu 24.04 LTS image/source for the new 2-core/2-GB server, with Docker already installed and the `ubuntu` user available
- operator names, maintenance window, rollback/abandonment condition, and emergency contact
- DNS zone owner and current `A`/`AAAA`/CAA records for `zarinpal.amiirkhl.ir`
- SSH source allowlist and at least two tested operator public keys
- confirmation that the trusted `ubuntu` user has the required Docker-group access
- approved deployment, backend, and frontend repository URLs and expected nested paths
- object-storage bucket/endpoint, retention capability, backup credentials, and separately escrowed encryption key
- target `main` branches and an empty location for the first three-hash release record
- secret inventory and owner; never put values in the ticket, shell history, or this repository

Do not change DNS before the host passes local and direct-IP validation. Do not reuse Qlockify credentials, certificates, volumes, backup paths, or Git credentials.

## Phase 1: establish recoverable access

1. Open the provider console/recovery channel and confirm it works independently of SSH.
2. Log in using the provider's initial method. Create named operator accounts and install only approved public keys.
3. Confirm a second operator can log in in a new session before changing SSH configuration.
4. Disable direct root SSH and password authentication only after key access and recovery console are proven.
5. Restrict SSH to the operator source allowlist at the provider firewall and host firewall. Keep the current session open while testing a new one.
6. Configure time synchronization and verify UTC plus the documented `Asia/Tehran` application assumption. The host itself should log in UTC.
7. Apply all supported security updates and reboot if required. Re-test console and SSH access after reboot.

Evidence: OS release/kernel, update result, time-sync state, two successful key logins, failed password/root-login test, and firewall rules. Do not record public key private material or session output containing secrets.

## Phase 2: harden and size the host

1. Set a unique hostname and record the provider instance id alongside it.
2. Configure default-deny inbound firewall rules. Initially allow SSH from the operator allowlist and TCP 80/443; verify from both an allowed and disallowed external source.
3. Enable unattended security updates with a documented reboot policy and alert path.
4. Configure disk/inode monitoring, time sync, audit/system logs, log rotation, and OOM visibility before application deployment.
5. Create the approved Qlockify-style runtime paths with explicit `ubuntu` ownership:

   ```text
   /home/ubuntu/zarinpal-analytics-deployment/           deployment repository and root .env
   /home/ubuntu/zarinpal-analytics-deployment/backend/   ignored backend checkout and backend .env
   /home/ubuntu/zarinpal-analytics-deployment/frontend/  ignored frontend checkout and frontend .env
   /home/ubuntu/zarinpal-analytics-deployment/releases/  ignored non-secret release records
   /home/ubuntu/zarinpal-analytics-deployment/backups/   ignored encrypted local backup retention
   ```

6. Verify every resolved absolute path is on the intended filesystem, not a symlink to a broad/unexpected target, before applying ownership or any future recursive operation.
7. Reserve enough free space for PostgreSQL growth, one temporary logical dump, three encrypted local backups, container images, and logs while retaining the platform alert floor of at least 20% and 10 GiB free.
8. Create and enable the approved 2 GB swap file, use restrictive permissions, persist it in `/etc/fstab`, and verify it after reboot.

## Phase 3: install and constrain Docker

1. Verify the preinstalled Docker Engine and Compose plugin came from Docker's approved Ubuntu repository; update/install only if verification fails, then record installed versions.
2. Confirm only the trusted `ubuntu` user has the required `docker` group access. Record that this access is effectively root-equivalent and restrict the account accordingly.
3. Configure the daemon's bounded logging policy before starting application containers.
4. Ensure Docker starts after reboot and that no unauthenticated TCP daemon socket is enabled.
5. Pull a harmless pinned public test image, verify its tag/digest, run it without published ports, then remove the test container.
6. Confirm the `ubuntu` user can run Docker Compose without using a broader shared administrator account.

Evidence: Engine/Compose versions, daemon security configuration, socket/group ownership, deploy-user membership, log driver/options, and test-image result.

## Phase 4: install the deployment control plane

1. From the `ubuntu` account, clone the deployment repository into `/home/ubuntu/zarinpal-analytics-deployment`, then clone backend and frontend into its exact ignored nested paths. Enter the source-forge HTTPS username/password interactively; do not configure a credential helper, token, or deploy key.
2. Confirm no Git credential was persisted in repository configuration, shell scripts, environment files, or the operating-system credential store.
3. Validate ownership, remote URL, `main` branch, current SHA, clean status, and independent `.git` directory for every repository.
4. Create the Qlockify-style Git-ignored environment files: deployment-root `.env` plus nested backend/frontend `.env`, all owned by `ubuntu`, mode 0600, and excluded from backup. Verify consumers receive only required values.
5. Manually run `git pull --ff-only origin main` in each repository, entering HTTPS credentials interactively, and confirm each checkout equals `origin/main`; do not reset, build, or start yet.
6. Record the three initial commit hashes and selected infrastructure image tags in a non-secret draft release record.
7. Render/validate Compose configuration without printing secret values. Confirm only Nginx publishes 80/443.

Do not run `docker compose up`, migrations, import, credential generation, restore, or DNS cutover during this phase.

## Phase 5: DNS and TLS staging

1. Lower DNS TTL in advance if an existing record is moving. Preserve the previous values and rollback time.
2. Point `A` (and `AAAA` only if IPv6 is actually routed/firewalled) for `zarinpal.amiirkhl.ir` after direct-IP Nginx/host validation. No `www` or alternate hostname is configured.
3. Verify public authoritative DNS from multiple resolvers and confirm the address matches the current server.
4. Start only the minimum edge services required for ACME HTTP-01. Verify the challenge path from an external network.
5. Obtain a staging certificate first and exercise renewal. Then request the production certificate.
6. Validate certificate names, chain, expiry, TLS 1.2/1.3, canonical redirects, unknown-host rejection, and renewal permissions.
7. Enable HSTS only after successful production issuance and renewal evidence; preload remains a separate approval.

If issuance fails, stop repeated production requests, inspect DNS/CAA/firewall/challenge routing, and continue serving no plaintext application content.

## Phase 6: first database, dataset, and application release

1. Acquire the production deployment lock and record the operator/release id.
2. Place the approved `challenge_data.csv.gz` at the exact Git-ignored deployment `data/incoming/` path. Verify regular-file/read-only properties, size, and pinned SHA-256; do not copy it into an image.
3. Invoke tracked `scripts/first-start.sh`. It starts PostgreSQL and Redis, waits with bounded backoff, applies the one-off migration, runs the one-shot importer with a read-only mount, and reconciles the pinned counts before starting public services.
4. Review the JSON/Markdown import report and durable import-run/issues metadata. Fatal validation, resource, migration, or reconciliation failure must exit nonzero before backend/frontend/Nginx activation.
5. Verify backend and worker liveness/readiness, worker heartbeat, Asynq `v0.26.0` compatibility, queue limits, and absence of unintended published ports.
6. Verify the frontend static sentinel plus the emitted Iran Yekan font resource. The source font/license remain frontend-owned; no deployment-side injection or copy occurs.
7. Verify Nginx public generic health, canonical redirect, security headers, login page, SSE behavior, and that only 80/443 are public.
8. Run demo credential generation as a separate explicit administrative job. Store one-time plaintext output outside Git/logs/backups.
9. Authenticate as at least two merchant accounts, verify different dashboards and cross-tenant isolation, then remove or escrow plaintext credential output.
10. Verify MVP operator checks: Compose health/restarts, structured logs, disk, certificate, queue/Redis state, and backup destination access.
11. Record the active three-hash release only after all postflight checks pass.

Failure before the active-release switch restores the prior/empty known state. Do not improvise a database down migration.

## Phase 7: bootstrap evidence review

1. Confirm the sole completed import run matches the pinned checksum, schema, source/session/attempt/merchant/terminal/category counts, and contains no fatal issue.
2. Confirm no raw staging remains and the ordinary `docker compose up -d` path cannot invoke migration/import/admin profiles.
3. Run `first-start.sh` a second time only in the controlled acceptance rehearsal and confirm the identical dataset is a safe no-op rather than another promotion.
4. Archive only safe non-sensitive release/import evidence; exclude the CSV, raw rows, reports containing restricted values, and credentials.

Never add the source archive or credentials to Git, a container image, backup package, or diagnostic bundle.

## Phase 8: backup and handoff gate

1. Run the first encrypted logical PostgreSQL backup.
2. Verify package contents, plain-SQL header/checksum, encrypted local artifact, remote upload, remote checksum, manifest, and retention counts.
3. Restore that backup to a clean non-production destination and complete application/tenant/analytics validation within the two-hour RTO.
4. Perform and record the MVP manual certificate, disk, backup-age, availability, API error, SSE heartbeat, and queue-age checks.
5. Record current release, DNS, certificate, backup id, achieved RPO/RTO, remaining exceptions, and on-call ownership.
6. Remove temporary bootstrap credentials/files and close provider rules that are no longer needed.

The host is not considered MVP-ready until restore rehearsal and every manual operator check passes. Prometheus/Grafana and automated alert delivery are the required next phase before AI work.

## Stop/rollback conditions

Stop without progressing when:

- recovery console or second-operator SSH access is not proven
- resolved paths, host identity, repository remotes/branches, environment, or release record do not match
- a repository is dirty/diverged unexpectedly or a build/source dependency cannot be verified
- unexpected ports are externally reachable
- DNS points somewhere unexpected or certificate names/chain are invalid
- migration, readiness, authenticated smoke, tenant isolation, worker/SSE, manual operational checks, backup, or restore rehearsal fails
- secrets appear in output, files have broad permissions, or the deployment checkout is dirty unexpectedly

Rollback DNS to the recorded prior values when the new host cannot pass the cutover window. Preserve logs and manifests, revoke any exposed credential, and do not delete the failed host until evidence and required backups are secured.

## Completion evidence

- [ ] Provider/host identity and capacity recorded
- [ ] Console plus two key-only operator paths proven
- [ ] Firewall external tests show only approved ports
- [ ] Docker/Compose versions and privilege model recorded
- [ ] Three repository remotes/branches/SHAs and release record verified
- [ ] Secrets outside Git with restrictive ownership/mode
- [ ] TLS issuance and renewal tested
- [ ] Migration, API, worker, frontend/font, Nginx, SSE, and tenant smoke pass
- [ ] One-time import reconciles and demo credentials stay outside logs/Git
- [ ] MVP manual health/log/disk/certificate/backup/queue checks pass
- [ ] Encrypted off-host backup and clean restore rehearsal pass within RTO
- [ ] Active/previous release ids and operator handoff recorded