DataOps-platform/docs/superpowers/plans/2026-07-30-wp13-offline-deployment-backup-recovery.md

WP13 Offline Deployment, Backup, and Recovery Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [x]) syntax for tracking.

Goal: Deliver a checksum-verifiable Compose offline package, upgrade preflight, consistent backup, guarded restore, and rollback rehearsal for the DataOps first-phase demonstration environment.

Architecture: app/, database/, and migrations/ remain the only development sources of truth and are synchronised into deployment/ before packaging. A small Bash operations layer wraps Docker Compose without embedding production secrets: it builds an image archive and release manifest on the connected build host, validates the bundle on the offline host, snapshots all Compose-managed volumes during a bounded maintenance window, and restores only after checksum and explicit target confirmation. The recovery rehearsal uses another Compose project and a separate port environment so the source volumes are never overwritten.

Tech Stack: Bash 3.2+, Docker Engine, Docker Compose v2, PostgreSQL/pgvector 16, SHA-256, tar/gzip, pytest contract tests.

---

Task 1: Define the WP13 delivery contract

Files:

• Create: tests/test_wp13_delivery_contract.py
• Modify: deploy/docker/docker-compose.yml

• Modify: deploy/docker/.env.example

• [x] Step 1: Write failing contract tests

Add tests that require:

• every locally built long-running service to have a stable image: name;
• every published port to support an environment override while retaining the existing local default;
• deployment/app, deployment/database, and deployment/migrations to match their source-of-truth trees byte-for-byte, excluding Python caches;
• the WP13 scripts and delivery documentation listed below to exist and expose --help;

• restore to reject a missing or mismatched confirmation token.

• [x] Step 2: Run the tests and verify RED

Run:

PYTHONPATH=. .venv/bin/pytest -q tests/test_wp13_delivery_contract.py

Expected: failures for release-copy drift, missing stable image declarations, and missing scripts.

• Step 3: Add stable image and port declarations

Give backend, runner, frontend, MCP, and LightRAG projector images stable local names. Replace fixed host ports with ${NAME:-default} expressions so a recovery project can use isolated ports without changing the canonical Compose file.

• Step 4: Synchronise the release copy

Run:

bash deployment/sync_release.sh

Remove generated Python caches from the release copy, then verify all three source trees match.

• Step 5: Re-run the focused contract subset

Run:

PYTHONPATH=. .venv/bin/pytest -q \
  tests/test_wp13_delivery_contract.py \
  tests/runner/test_compose_contract.py \
  tests/mcp/test_deployment_contract.py

Expected: only tests for not-yet-created WP13 scripts remain failing.

Task 2: Build the offline Compose package and preflight

Files:

• Create: deployment/compose/wp13-common.sh
• Create: deployment/compose/preflight.sh
• Create: deployment/compose/install_offline.sh
• Create: deployment/compose/package_offline.sh
• Create: deployment/compose/README.md
• Modify: deployment/package_release.sh

• Modify: deployment/MANIFEST.md

• [x] Step 1: Extend tests for package contents and checksum failure

Use a temporary output directory and a no-image test mode. Require the package to contain the Compose definition, bind-mounted configuration, database migrations, .env.example, install/preflight scripts, release-manifest.txt, and checksums.sha256. Mutating one packaged file must make preflight fail before any Docker mutation.

• Step 2: Run the package tests and verify RED

Run the two named package tests and confirm the failure is caused by missing commands.

• Step 3: Implement common verification and preflight

The common layer must:

• resolve repository/package paths without relying on the caller working directory;
• provide portable SHA-256 generation and verification for macOS and Linux;
• invoke Docker Compose with an explicit project, Compose file, and optional environment file;
• reject unreadable archives and unresolved Compose configuration;

• compare Alembic current/head during upgrade checks and never run a downgrade.

• [x] Step 4: Implement offline packaging

package_offline.sh must copy only runtime-required files, record the Git commit and image IDs, save all default-profile images into images/dataops-images.tar unless --skip-images is used by contract tests, generate checksums, and create dataops-platform-offline-<version>.tar.gz.

• Step 5: Implement guarded installation

install_offline.sh must verify checksums, load images, run preflight, start with --no-build, wait for Compose health, and record the installed manifest. It must not print environment values or create an administrator.

• Step 6: Run focused package tests

Run:

PYTHONPATH=. .venv/bin/pytest -q tests/test_wp13_delivery_contract.py -k 'package or preflight or install'

Expected: PASS.

Task 3: Implement consistent backup, restore, and rollback

Files:

• Create: deployment/compose/backup.sh
• Create: deployment/compose/restore.sh
• Create: deployment/compose/rollback.sh
• Modify: deployment/compose/README.md

• Modify: tests/test_wp13_delivery_contract.py

• [x] Step 1: Write failing safety and manifest tests

Require backup to name every Compose-managed volume, include a logical PostgreSQL export and Alembic revision, and checksum every payload. Require restore to fail before mutation when checksums fail, the destination project is already running, the destination is the source project, or --confirm-project does not exactly match.

• Step 2: Run the restore safety tests and verify RED

Run:

PYTHONPATH=. .venv/bin/pytest -q tests/test_wp13_delivery_contract.py -k 'backup or restore or rollback'

Expected: failures for missing backup and restore commands.

• Step 3: Implement bounded cold backup

Create a logical PostgreSQL export first, stop the source project for a consistent volume snapshot, archive every volume discovered by Compose labels, restart the source project through a trap, write a non-secret manifest, and checksum all payloads. Refuse to proceed when the project is unhealthy unless --allow-degraded is explicit.

• Step 4: Implement guarded restore

Verify checksums and manifest first, require an exact new project confirmation, refuse existing/running destinations, create labelled destination volumes, restore each archive, then start Compose with --no-build. A same-project restore remains disabled by default.

• Step 5: Implement rollback wrapper

Rollback must be an explicit restore of a named pre-upgrade backup; it may roll back application images and persisted state together but must never call alembic downgrade.

• Step 6: Re-run focused tests

Run the complete WP13 contract test and the existing Compose contracts.

Task 4: Perform an isolated recovery rehearsal

Files:

• Create: deployment/compose/.env.rehearsal.example

• Create: docs/validation/WP13_DEPLOYMENT_BACKUP_RECOVERY_EVIDENCE.md

• [x] Step 1: Run upgrade preflight on the current stack

Record Docker/Compose versions, clean release-copy status, required image availability, Compose resolution, Alembic current/head, and source health without exposing credentials.

• Step 2: Create a real local backup

Use a temporary directory outside Git, accept the bounded maintenance window, and record backup size, payload count, manifest hash, and restart health.

• Step 3: Restore under an isolated project and port set

Use project dataops-wp13-restore and the rehearsal environment port overrides. Do not allow in-place restore.

• Step 4: Verify recovered evidence

Verify:

• the recovered PostgreSQL Alembic revision is at head;
• governance tables and WP12 seal records exist;
• Neo4j and MinIO become healthy;
• backend and frontend become healthy;

• source and recovered volume names are disjoint.

• [x] Step 5: Remove only the rehearsal project

Resolve containers and volumes by the exact dataops-wp13-restore Compose label, stop the rehearsal project, and remove only its temporary volumes. Confirm the original dataops-test project is still healthy.

Task 5: Close WP13 documentation and status

Files:

• Modify: docs/DATAOPS_PHASE1_3_MONTH_WORK_PLAN_20260729.md
• Modify: docs/FUNCTION_MODULE_CENSUS_20260726.md
• Modify: deployment/README.md
• Modify: deployment/DEPLOYMENT_GUIDE.md
• Modify: deployment/CHECKLIST.md
• Modify: deployment/MANIFEST.md

• Create: docs/validation/WP13_DEPLOYMENT_BACKUP_RECOVERY_EVIDENCE.md

• [x] Step 1: Record evidence and boundaries

Distinguish local engineering completion from enterprise acceptance. Keep enterprise image-signing/notarisation, external backup storage, key custody, production RPO/RTO, real clean host, disaster-domain separation, and production change approval as remaining gates.

• Step 2: Update maturity conservatively

Only advance ledger items directly evidenced by the WP13 scripts and rehearsal. Do not mark SSO, production disaster recovery, external WORM storage, or production release complete.

• Step 3: Run final targeted verification

Run:

PYTHONPATH=. .venv/bin/pytest -q \
  tests/test_wp13_delivery_contract.py \
  tests/runner/test_compose_contract.py \
  tests/mcp/test_deployment_contract.py \
  tests/test_database_migrations.py
bash -n deployment/compose/*.sh
docker compose -f deploy/docker/docker-compose.yml config --quiet

Also inspect the generated package checksums, backup checksums, recovery report, release-copy diff, Git diff, and working-tree scope.

• Step 4: Commit locally

Stage only WP13 files and synchronised release-copy changes, then create:

git commit -m "feat: add offline deployment and recovery workflow"

Do not push or deploy to production.