# WP13 Offline Deployment, Backup, and Recovery Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [x]`) syntax for tracking. **Goal:** Deliver a checksum-verifiable Compose offline package, upgrade preflight, consistent backup, guarded restore, and rollback rehearsal for the DataOps first-phase demonstration environment. **Architecture:** `app/`, `database/`, and `migrations/` remain the only development sources of truth and are synchronised into `deployment/` before packaging. A small Bash operations layer wraps Docker Compose without embedding production secrets: it builds an image archive and release manifest on the connected build host, validates the bundle on the offline host, snapshots all Compose-managed volumes during a bounded maintenance window, and restores only after checksum and explicit target confirmation. The recovery rehearsal uses another Compose project and a separate port environment so the source volumes are never overwritten. **Tech Stack:** Bash 3.2+, Docker Engine, Docker Compose v2, PostgreSQL/pgvector 16, SHA-256, tar/gzip, pytest contract tests. --- ### Task 1: Define the WP13 delivery contract **Files:** - Create: `tests/test_wp13_delivery_contract.py` - Modify: `deploy/docker/docker-compose.yml` - Modify: `deploy/docker/.env.example` - [x] **Step 1: Write failing contract tests** Add tests that require: - every locally built long-running service to have a stable `image:` name; - every published port to support an environment override while retaining the existing local default; - `deployment/app`, `deployment/database`, and `deployment/migrations` to match their source-of-truth trees byte-for-byte, excluding Python caches; - the WP13 scripts and delivery documentation listed below to exist and expose `--help`; - restore to reject a missing or mismatched confirmation token. - [x] **Step 2: Run the tests and verify RED** Run: ```bash PYTHONPATH=. .venv/bin/pytest -q tests/test_wp13_delivery_contract.py ``` Expected: failures for release-copy drift, missing stable image declarations, and missing scripts. - [x] **Step 3: Add stable image and port declarations** Give backend, runner, frontend, MCP, and LightRAG projector images stable local names. Replace fixed host ports with `${NAME:-default}` expressions so a recovery project can use isolated ports without changing the canonical Compose file. - [x] **Step 4: Synchronise the release copy** Run: ```bash bash deployment/sync_release.sh ``` Remove generated Python caches from the release copy, then verify all three source trees match. - [x] **Step 5: Re-run the focused contract subset** Run: ```bash PYTHONPATH=. .venv/bin/pytest -q \ tests/test_wp13_delivery_contract.py \ tests/runner/test_compose_contract.py \ tests/mcp/test_deployment_contract.py ``` Expected: only tests for not-yet-created WP13 scripts remain failing. ### Task 2: Build the offline Compose package and preflight **Files:** - Create: `deployment/compose/wp13-common.sh` - Create: `deployment/compose/preflight.sh` - Create: `deployment/compose/install_offline.sh` - Create: `deployment/compose/package_offline.sh` - Create: `deployment/compose/README.md` - Modify: `deployment/package_release.sh` - Modify: `deployment/MANIFEST.md` - [x] **Step 1: Extend tests for package contents and checksum failure** Use a temporary output directory and a no-image test mode. Require the package to contain the Compose definition, bind-mounted configuration, database migrations, `.env.example`, install/preflight scripts, `release-manifest.txt`, and `checksums.sha256`. Mutating one packaged file must make preflight fail before any Docker mutation. - [x] **Step 2: Run the package tests and verify RED** Run the two named package tests and confirm the failure is caused by missing commands. - [x] **Step 3: Implement common verification and preflight** The common layer must: - resolve repository/package paths without relying on the caller working directory; - provide portable SHA-256 generation and verification for macOS and Linux; - invoke Docker Compose with an explicit project, Compose file, and optional environment file; - reject unreadable archives and unresolved Compose configuration; - compare Alembic current/head during upgrade checks and never run a downgrade. - [x] **Step 4: Implement offline packaging** `package_offline.sh` must copy only runtime-required files, record the Git commit and image IDs, save all default-profile images into `images/dataops-images.tar` unless `--skip-images` is used by contract tests, generate checksums, and create `dataops-platform-offline-.tar.gz`. - [x] **Step 5: Implement guarded installation** `install_offline.sh` must verify checksums, load images, run preflight, start with `--no-build`, wait for Compose health, and record the installed manifest. It must not print environment values or create an administrator. - [x] **Step 6: Run focused package tests** Run: ```bash PYTHONPATH=. .venv/bin/pytest -q tests/test_wp13_delivery_contract.py -k 'package or preflight or install' ``` Expected: PASS. ### Task 3: Implement consistent backup, restore, and rollback **Files:** - Create: `deployment/compose/backup.sh` - Create: `deployment/compose/restore.sh` - Create: `deployment/compose/rollback.sh` - Modify: `deployment/compose/README.md` - Modify: `tests/test_wp13_delivery_contract.py` - [x] **Step 1: Write failing safety and manifest tests** Require backup to name every Compose-managed volume, include a logical PostgreSQL export and Alembic revision, and checksum every payload. Require restore to fail before mutation when checksums fail, the destination project is already running, the destination is the source project, or `--confirm-project` does not exactly match. - [x] **Step 2: Run the restore safety tests and verify RED** Run: ```bash PYTHONPATH=. .venv/bin/pytest -q tests/test_wp13_delivery_contract.py -k 'backup or restore or rollback' ``` Expected: failures for missing backup and restore commands. - [x] **Step 3: Implement bounded cold backup** Create a logical PostgreSQL export first, stop the source project for a consistent volume snapshot, archive every volume discovered by Compose labels, restart the source project through a trap, write a non-secret manifest, and checksum all payloads. Refuse to proceed when the project is unhealthy unless `--allow-degraded` is explicit. - [x] **Step 4: Implement guarded restore** Verify checksums and manifest first, require an exact new project confirmation, refuse existing/running destinations, create labelled destination volumes, restore each archive, then start Compose with `--no-build`. A same-project restore remains disabled by default. - [x] **Step 5: Implement rollback wrapper** Rollback must be an explicit restore of a named pre-upgrade backup; it may roll back application images and persisted state together but must never call `alembic downgrade`. - [x] **Step 6: Re-run focused tests** Run the complete WP13 contract test and the existing Compose contracts. ### Task 4: Perform an isolated recovery rehearsal **Files:** - Create: `deployment/compose/.env.rehearsal.example` - Create: `docs/validation/WP13_DEPLOYMENT_BACKUP_RECOVERY_EVIDENCE.md` - [x] **Step 1: Run upgrade preflight on the current stack** Record Docker/Compose versions, clean release-copy status, required image availability, Compose resolution, Alembic current/head, and source health without exposing credentials. - [x] **Step 2: Create a real local backup** Use a temporary directory outside Git, accept the bounded maintenance window, and record backup size, payload count, manifest hash, and restart health. - [x] **Step 3: Restore under an isolated project and port set** Use project `dataops-wp13-restore` and the rehearsal environment port overrides. Do not allow in-place restore. - [x] **Step 4: Verify recovered evidence** Verify: - the recovered PostgreSQL Alembic revision is at head; - governance tables and WP12 seal records exist; - Neo4j and MinIO become healthy; - backend and frontend become healthy; - source and recovered volume names are disjoint. - [x] **Step 5: Remove only the rehearsal project** Resolve containers and volumes by the exact `dataops-wp13-restore` Compose label, stop the rehearsal project, and remove only its temporary volumes. Confirm the original `dataops-test` project is still healthy. ### Task 5: Close WP13 documentation and status **Files:** - Modify: `docs/DATAOPS_PHASE1_3_MONTH_WORK_PLAN_20260729.md` - Modify: `docs/FUNCTION_MODULE_CENSUS_20260726.md` - Modify: `deployment/README.md` - Modify: `deployment/DEPLOYMENT_GUIDE.md` - Modify: `deployment/CHECKLIST.md` - Modify: `deployment/MANIFEST.md` - Create: `docs/validation/WP13_DEPLOYMENT_BACKUP_RECOVERY_EVIDENCE.md` - [x] **Step 1: Record evidence and boundaries** Distinguish local engineering completion from enterprise acceptance. Keep enterprise image-signing/notarisation, external backup storage, key custody, production RPO/RTO, real clean host, disaster-domain separation, and production change approval as remaining gates. - [x] **Step 2: Update maturity conservatively** Only advance ledger items directly evidenced by the WP13 scripts and rehearsal. Do not mark SSO, production disaster recovery, external WORM storage, or production release complete. - [x] **Step 3: Run final targeted verification** Run: ```bash PYTHONPATH=. .venv/bin/pytest -q \ tests/test_wp13_delivery_contract.py \ tests/runner/test_compose_contract.py \ tests/mcp/test_deployment_contract.py \ tests/test_database_migrations.py bash -n deployment/compose/*.sh docker compose -f deploy/docker/docker-compose.yml config --quiet ``` Also inspect the generated package checksums, backup checksums, recovery report, release-copy diff, Git diff, and working-tree scope. - [x] **Step 4: Commit locally** Stage only WP13 files and synchronised release-copy changes, then create: ```bash git commit -m "feat: add offline deployment and recovery workflow" ``` Do not push or deploy to production.