DataOps-platform/docs/acceptance/data-factory/20260724-task9-remediation-summary.md

Data Factory Task 9 remediation acceptance

Date: 2026-07-24 Baseline commit: 4ce301a Migration: 20260724_240 -> 20260724_250

Safety outcomes

• A PostgreSQL lease keyed by (dataflow_uid, environment) permits exactly one in-flight external mutation across deployments, actions and idempotency keys.
• Every operation records an owner token, lease expiry, attempt epoch and monotonic fencing epoch. Finalization, failure, unknown-state recording and compensation are fenced against the expected owner.
• Expired claimed operations can be reclaimed. Deployment, activation and rollback reconciliation observe Kestra state and complete only missing idempotent work; rollback recovery covers a process exit between candidate disable and prior-version enable.
• The stored definition digest is computed from the canonical real Kestra flow returned by GET /flows/{namespace}/{id}. Canary, activation, rollback and reconciliation fail closed when the real definition drifts.
• Definition attestations are explicitly versioned as yaml_v1 or kestra_canonical_v2. Existing yaml_v1 deployments remain operable only through a controlled real-flow re-attestation; any evidence linked to the old digest is invalidated and a new canary is required before activation.
• Legacy revision-240 operations left as claimed without a complete lease/fencing tuple are migrated to unknown with legacy_claim_requires_reconciliation; their request, actor and idempotency key remain intact for operator reconciliation.
• Any unresolved unknown operation is a hard barrier for the complete (dataflow_uid, environment) scope. A different deployment, action or idempotency key receives the oldest unresolved operation as its explicit reconciliation target and cannot create a new claim or external call. Multiple unknowns are drained deterministically oldest-first, with the response indicating when more than one remains.
• A successful Kestra create response is not trusted by itself. The immediate follow-up GET must match the compiled flow ID, namespace, governance labels and canonical executable definition before its server-observed hash can be persisted.
• Deployment admission builds a separate server-owned physical_workflow_spec: every rule.apply and quality.check node is mapped one-to-one by component_binding_id to its published physical plan_hash. Missing, duplicate or extra mappings fail closed. The released semantic package and package_hash remain unchanged, while the deployment stores and re-hashes only the physical workflow.
• Output bindings strictly accept write or read_write and still require write_mode; inputs remain strictly read. Canary task tokens authorize writes only for nodes whose governed purpose is write, so quality nodes remain read-only.
• API conflicts expose operation_in_progress, operation_unknown and terminal_conflict. The browser retains the operation key for the first two and rotates it only for an explicit terminal conflict.

Automated verification

• Fail-first remediation scenarios: 6 failures before implementation; all passed after implementation.
• Recovery-window tests prove that a reclaimed deploy with no flow performs exactly one deploy, a reclaimed activation of a disabled flow performs exactly one enable, and rollback resumes correctly from all three observable external states (zero, one or two mutations completed).
• Focused repository/backend/API/frontend/schema/PostgreSQL/Kestra acceptance: 114 passed.
• Full Python suite: 735 passed, 36 skipped, 59 subtests passed.
• Ruff over changed Python: passed.
• git diff --check: passed.
• Frontend production image build and rebuilt backend/Runner images: passed with no errors; pre-existing frontend warning classes remain.
• PostgreSQL was upgraded from real revision 20260724_240 to 20260724_250; the integration test verifies competing-operation fencing and expired-claim recovery with a higher fencing epoch.
• A separate database was migrated from zero through revision 240, seeded with a real legacy incomplete claim, then upgraded through revision 250. The operation became unknown, retained its request/actor/key, and the database was removed after verification.
• Real PostgreSQL acceptance also proves that an unknown result with no active lease blocks a new key on a different action and deployment, returns the original operation, and leaves no new ledger row.

Ruff was run over the complete changed-Python scope, including migration 250:

app/api/data_rules/routes.py app/core/data_rules/deployment.py app/core/data_rules/repository.py app/core/orchestration/compilers/kestra.py app/core/orchestration/engines/kestra.py migrations/versions/20260724_250_data_factory_operation_leases.py tests/core/data_rules/test_deployment.py tests/core/data_rules/test_data_rule_repository.py tests/integration/test_data_factory_postgres_lifecycle.py tests/test_data_rule_api.py tests/test_data_rule_frontend_contract.py tests/test_data_rule_schema.py tests/test_kestra_local_contract.py

30-minute Kestra queue/restart soak

• Command used the script's release default: KESTRA_SOAK_SECONDS=1800.
• Start: 2026-07-23T21:16:32Z.
• End: 2026-07-23T21:46:29Z.
• Real executions: 111.
• Successful terminal executions: 111.
• Failed/non-success terminal executions: 0.
• Samples before controlled PostgreSQL/Kestra restart: 38.
• Samples after both services recovered: 73.
• Every health sample: UP.
• Evidence: docs/acceptance/data-factory/kestra-queue-soak-30m.jsonl.
• Evidence SHA-256: 5eedb97bc96955359c09bf9764edd0742e65ebd4f8bab7f3a2c660f1a48d317b.

The local backend, frontend, Runner, PostgreSQL and Kestra containers were healthy after the soak. The backend health endpoint returned application code 200 with database and Neo4j healthy.

Revised soak-harness verification

The soak harness now records the observation-window start, full Docker log window, and PostgreSQL/Kestra RestartCount before and after the controlled restart. The controlled docker restart does not increment RestartCount, so any delta identifies an uncontrolled policy/daemon restart and fails the gate.

A bounded 60-second verification of the revised harness completed successfully:

• Start: 2026-07-23T22:05:38Z.
• End: 2026-07-23T22:06:42Z.
• Real executions: 8; successful: 8; failed: 0.
• Samples after the controlled restart: 4.
• PostgreSQL and Kestra RestartCount: 0 -> 0.
• Full-window known controlled disconnect signatures: 28.
• Evidence: docs/acceptance/data-factory/kestra-queue-soak-script-v2-60s.jsonl.
• Evidence SHA-256: a736d90242f896b751e5cf697426c24053cf1feb9a0970cae99c8d8dc2bb2f7f.

The first default-duration v2 attempt retained all 110 successful samples but correctly has no soak_complete: the then-current final scanner treated PostgreSQL SQLSTATE 57P01 (terminating connection due to administrator command) from the deliberate restart as an unexpected fatal signature. That failed evidence remains unchanged at docs/acceptance/data-factory/kestra-queue-soak-v2-30m.jsonl.

The scanner was corrected to retain and count that exact controlled-restart signature while every other FATAL or OOM signature remains fail-closed. A second default-duration run then completed the full release gate:

• Start: 2026-07-23T22:58:52Z.
• End: 2026-07-23T23:29:03Z.
• Real executions and successes: 111/111.
• Samples before restart: 38; after recovery: 73.
• PostgreSQL and Kestra RestartCount: 0 -> 0.
• Known controlled restart disconnect signatures: 18.
• Full log window: 2026-07-23T22:58:52Z through completion.
• Evidence: docs/acceptance/data-factory/kestra-queue-soak-v2-30m-rerun.jsonl.
• Evidence SHA-256: e57b161428b78afaba12c14ef4ef2a12ff098e5a447d4cb4204fdace2d72dffa.