# Data Factory Task 9 remediation acceptance Date: 2026-07-24 Baseline commit: `4ce301a` Migration: `20260724_240 -> 20260724_250` ## Safety outcomes - A PostgreSQL lease keyed by `(dataflow_uid, environment)` permits exactly one in-flight external mutation across deployments, actions and idempotency keys. - Every operation records an owner token, lease expiry, attempt epoch and monotonic fencing epoch. Finalization, failure, unknown-state recording and compensation are fenced against the expected owner. - Expired `claimed` operations can be reclaimed. Deployment, activation and rollback reconciliation observe Kestra state and complete only missing idempotent work; rollback recovery covers a process exit between candidate disable and prior-version enable. - The stored definition digest is computed from the canonical real Kestra flow returned by `GET /flows/{namespace}/{id}`. Canary, activation, rollback and reconciliation fail closed when the real definition drifts. - Definition attestations are explicitly versioned as `yaml_v1` or `kestra_canonical_v2`. Existing `yaml_v1` deployments remain operable only through a controlled real-flow re-attestation; any evidence linked to the old digest is invalidated and a new canary is required before activation. - Legacy revision-240 operations left as `claimed` without a complete lease/fencing tuple are migrated to `unknown` with `legacy_claim_requires_reconciliation`; their request, actor and idempotency key remain intact for operator reconciliation. - Any unresolved `unknown` operation is a hard barrier for the complete `(dataflow_uid, environment)` scope. A different deployment, action or idempotency key receives the oldest unresolved operation as its explicit reconciliation target and cannot create a new claim or external call. Multiple unknowns are drained deterministically oldest-first, with the response indicating when more than one remains. - A successful Kestra create response is not trusted by itself. The immediate follow-up `GET` must match the compiled flow ID, namespace, governance labels and canonical executable definition before its server-observed hash can be persisted. - Deployment admission builds a separate server-owned `physical_workflow_spec`: every `rule.apply` and `quality.check` node is mapped one-to-one by `component_binding_id` to its published physical `plan_hash`. Missing, duplicate or extra mappings fail closed. The released semantic package and `package_hash` remain unchanged, while the deployment stores and re-hashes only the physical workflow. - Output bindings strictly accept `write` or `read_write` and still require `write_mode`; inputs remain strictly `read`. Canary task tokens authorize writes only for nodes whose governed purpose is `write`, so quality nodes remain read-only. - API conflicts expose `operation_in_progress`, `operation_unknown` and `terminal_conflict`. The browser retains the operation key for the first two and rotates it only for an explicit terminal conflict. ## Automated verification - Fail-first remediation scenarios: 6 failures before implementation; all passed after implementation. - Recovery-window tests prove that a reclaimed deploy with no flow performs exactly one deploy, a reclaimed activation of a disabled flow performs exactly one enable, and rollback resumes correctly from all three observable external states (zero, one or two mutations completed). - Focused repository/backend/API/frontend/schema/PostgreSQL/Kestra acceptance: `114 passed`. - Full Python suite: `735 passed, 36 skipped, 59 subtests passed`. - Ruff over changed Python: passed. - `git diff --check`: passed. - Frontend production image build and rebuilt backend/Runner images: passed with no errors; pre-existing frontend warning classes remain. - PostgreSQL was upgraded from real revision `20260724_240` to `20260724_250`; the integration test verifies competing-operation fencing and expired-claim recovery with a higher fencing epoch. - A separate database was migrated from zero through revision 240, seeded with a real legacy incomplete claim, then upgraded through revision 250. The operation became `unknown`, retained its request/actor/key, and the database was removed after verification. - Real PostgreSQL acceptance also proves that an unknown result with no active lease blocks a new key on a different action and deployment, returns the original operation, and leaves no new ledger row. Ruff was run over the complete changed-Python scope, including migration 250: `app/api/data_rules/routes.py app/core/data_rules/deployment.py app/core/data_rules/repository.py app/core/orchestration/compilers/kestra.py app/core/orchestration/engines/kestra.py migrations/versions/20260724_250_data_factory_operation_leases.py tests/core/data_rules/test_deployment.py tests/core/data_rules/test_data_rule_repository.py tests/integration/test_data_factory_postgres_lifecycle.py tests/test_data_rule_api.py tests/test_data_rule_frontend_contract.py tests/test_data_rule_schema.py tests/test_kestra_local_contract.py` ## 30-minute Kestra queue/restart soak - Command used the script's release default: `KESTRA_SOAK_SECONDS=1800`. - Start: `2026-07-23T21:16:32Z`. - End: `2026-07-23T21:46:29Z`. - Real executions: `111`. - Successful terminal executions: `111`. - Failed/non-success terminal executions: `0`. - Samples before controlled PostgreSQL/Kestra restart: `38`. - Samples after both services recovered: `73`. - Every health sample: `UP`. - Evidence: `docs/acceptance/data-factory/kestra-queue-soak-30m.jsonl`. - Evidence SHA-256: `5eedb97bc96955359c09bf9764edd0742e65ebd4f8bab7f3a2c660f1a48d317b`. The local backend, frontend, Runner, PostgreSQL and Kestra containers were healthy after the soak. The backend health endpoint returned application code 200 with database and Neo4j healthy. ## Revised soak-harness verification The soak harness now records the observation-window start, full Docker log window, and PostgreSQL/Kestra `RestartCount` before and after the controlled restart. The controlled `docker restart` does not increment `RestartCount`, so any delta identifies an uncontrolled policy/daemon restart and fails the gate. A bounded 60-second verification of the revised harness completed successfully: - Start: `2026-07-23T22:05:38Z`. - End: `2026-07-23T22:06:42Z`. - Real executions: `8`; successful: `8`; failed: `0`. - Samples after the controlled restart: `4`. - PostgreSQL and Kestra `RestartCount`: `0 -> 0`. - Full-window known controlled disconnect signatures: `28`. - Evidence: `docs/acceptance/data-factory/kestra-queue-soak-script-v2-60s.jsonl`. - Evidence SHA-256: `a736d90242f896b751e5cf697426c24053cf1feb9a0970cae99c8d8dc2bb2f7f`. The first default-duration v2 attempt retained all `110` successful samples but correctly has no `soak_complete`: the then-current final scanner treated PostgreSQL SQLSTATE `57P01` (`terminating connection due to administrator command`) from the deliberate restart as an unexpected fatal signature. That failed evidence remains unchanged at `docs/acceptance/data-factory/kestra-queue-soak-v2-30m.jsonl`. The scanner was corrected to retain and count that exact controlled-restart signature while every other FATAL or OOM signature remains fail-closed. A second default-duration run then completed the full release gate: - Start: `2026-07-23T22:58:52Z`. - End: `2026-07-23T23:29:03Z`. - Real executions and successes: `111/111`. - Samples before restart: `38`; after recovery: `73`. - PostgreSQL and Kestra `RestartCount`: `0 -> 0`. - Known controlled restart disconnect signatures: `18`. - Full log window: `2026-07-23T22:58:52Z` through completion. - Evidence: `docs/acceptance/data-factory/kestra-queue-soak-v2-30m-rerun.jsonl`. - Evidence SHA-256: `e57b161428b78afaba12c14ef4ef2a12ff098e5a447d4cb4204fdace2d72dffa`.