DataOps-platform/docs/superpowers/plans/2026-07-16-dataflow-workflow-versioning.md

DataFlow n8n Workflow Versioning Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:test-driven-development and superpowers:executing-plans to implement this plan task-by-task.

Goal: Map each governed DataFlow to multiple identifiable n8n Workflow versions while enforcing one active version per environment.

Architecture: Neo4j remains the DataFlow definition store; PostgreSQL owns immutable workflow version mappings and activation state. The backend coordinates n8n API calls through a state machine and outbox, with a partial unique index as the final concurrency guard.

Tech Stack: Flask, PostgreSQL, Neo4j, n8n REST API, pytest, Vue 2.

---

Task 1: Create workflow version schema

Files:

• Create: migrations/versions/20260716_20_workflow_versions.py
• Create: app/core/data_flow/workflow_models.py

• Create: tests/test_workflow_version_schema.py

• [ ] Test version uniqueness and concurrent attempts to create two active rows for one (dataflow_uid, environment).

• [ ] Create dataflow_workflow_versions with UUID ID, stable dataflow_uid, environment enum, monotonically increasing version number, n8n ID, immutable definition hash, status and audit fields.

• [ ] Add UNIQUE(dataflow_uid, environment, version_no) and the partial unique active index.

• [ ] Seed no production mappings automatically; provide an import report command for existing n8n workflows.

Task 2: Add repository and n8n definition snapshotting

Files:

• Create: app/core/data_flow/workflow_repository.py
• Modify: app/core/data_factory/n8n_client.py
• Create: tests/test_workflow_repository.py

• Modify: tests/test_n8n_client_config.py

• [ ] Test creating a version from an n8n workflow, canonical JSON hashing, missing workflow, duplicate hash and redaction of credentials.

• [ ] Add n8n client methods for read/export/activate/deactivate with explicit timeouts and typed errors.

• [ ] Persist only a safe definition snapshot or hash; never store n8n credential secrets in PostgreSQL.

• [ ] Use stable DataFlow UID and reject numeric Neo4j IDs for new mappings.

Task 3: Implement activation state machine

Files:

• Create: app/core/data_flow/workflow_activation.py

• Create: tests/test_workflow_activation.py

• [ ] Test draft-to-validating-to-active, old active-to-superseded, n8n failure, retry, duplicate request and two concurrent activation requests.

• [ ] Lock the environment mapping, mark the target validating, invoke n8n, then atomically switch PostgreSQL active status and append outbox events.

• [ ] If n8n succeeds but PostgreSQL finalization fails, reconcile from n8n state and correlation ID instead of blindly reactivating.

• [ ] Expose activation_failed with actionable error metadata that excludes API keys.

Task 4: Add governed version APIs

Files:

• Modify: app/api/data_flow/routes.py
• Create: app/api/data_flow/schemas.py

• Create: tests/test_workflow_version_api.py

• [ ] Add list/create/detail/activate endpoints under /api/dataflow/{dataflow_uid}/workflow-versions.

• [ ] Require editor permission to create versions and administrator/editor-with-activation permission to activate.

• [ ] Return both DataFlow identity and n8n Workflow identity, version, environment, status and timestamps.

• [ ] Keep /api/datafactory/workflows/* for n8n-native diagnostics; do not rename Workflow globally.

Task 5: Build the version management UI

Files:

• Modify: frontend/src/views/dataGovernance/dataProcess/index.vue
• Create: frontend/src/views/dataGovernance/dataProcess/components/workflowVersions.vue
• Modify: frontend/src/api/dataGovernance.js

• Create: tests/e2e/workflow-versions.spec.js

• [ ] Display DataFlow details separately from n8n Workflow versions and show the active version per environment.

• [ ] Require confirmation before activation and disable actions while a request is in flight.

• [ ] Show validation/activation errors and n8n execution links without exposing credentials.

• [ ] Verify viewer read-only, editor version creation and authorized activation in browser tests.

Task 6: Add reconciliation and operational acceptance

Files:

• Create: app/commands/reconcile_workflow_versions.py
• Create: tests/integration/test_n8n_reconciliation.py

• Modify: deploy/docker/README.md

• [ ] Compare PostgreSQL active mappings with local n8n activation state and emit a report before making repairs.

• [ ] Test n8n downtime, request timeout, backend restart and out-of-band n8n deactivation.

• [ ] Run full local-stack acceptance and prove one active version per environment under concurrency.

• [ ] Regenerate OpenAPI and publish operator recovery commands.