# Task 6 Report — Artifact handoff and governed rule evidence

Date: 2026-07-23
Branch: `codex/data-rule-execution-m3a-m5`

Lifecycle boundary: Task 6 consumes only plans already marked `published`.
It does not add, relax, or emulate the Task 7 publication gate.

## Outcome

Task 6 now provides a production-wired path from a signed Runner task to one
server-attested `rule_runs` record, optional durable violation evidence, and a
closed artifact-only handoff to the next node.

The implementation includes:

- `PostgresRuleEvidenceWriter` with canonical deployment, component, rule
  version, plan hash, DataFlow, workflow version, node, and correlation
  attestation;
- a server-derived evidence key that makes a node execution idempotent across
  different signed task tokens and Runner retries;
- reliable `running` to `success`, `failed`, `unknown`, or `cancelled`
  finalization with row counts, timings, commit outcome, and bounded public
  result;
- a signed, exact deployment/environment task identity plus a durable lease
  owner, heartbeat, and expiry boundary; an expired run is finalized
  `unknown` and is never blindly re-executed;
- same-JTI response replay from the durable ledger, including recovery from
  terminal rule evidence when execution committed but the HTTP/ledger response
  was lost;
- commit-acknowledgement rechecks for both run and sample finalization;
- a maximum 100-row violation sample, redacted before it leaves the isolated
  Polars worker and redacted again at the evidence boundary;
- a `pending` to verified MinIO upload to `ready` sample state machine with
  digest, schema hash, reference, expiry, and bounded TTL cleanup;
- no raw sample values in Runner responses or application log calls;
- correlation-scoped upstream output resolution that accepts only current,
  ready, unexpired, storage-attested artifacts;
- an opaque `dataops-staging://<uuid>` SQL receipt for same-source pushdown
  handoff, bound to producer run, deployment, correlation, dataset binding,
  binding hash, relation digest, commit state, and expiry;
- bracket-safe Kestra single-predecessor `input_artifact` expressions;
  governed rule fan-in is rejected until an explicit merge component exists;
- a top-level Runner `output_artifact` field matching the Kestra expression
  contract while retaining the full bounded result object.

## Forward-only schema change

Migrations `20260723_160`, `20260723_170`, `20260723_180`, and
`20260723_190` extend the existing evidence tables without changing historical
migrations 110, 140, or 150. The second review did not rewrite already-applied
migrations 160 or 170, and the third review leaves migration 180 unchanged.
Each additional schema revision is isolated in its own forward-only migration.

`rule_runs` gains:

- nullable legacy-compatible `evidence_key` with a unique constraint;
- `commit_outcome`;
- bounded `public_result`;
- `failure_code` and `updated_at`.

`rule_violation_samples` gains:

- artifact and schema digests;
- `legacy`, `pending`, `ready`, and `failed` handoff states;
- failure and update fields;
- one sample per run;
- a strict 100-row ceiling and retention index.

Downgrade is deliberately rejected because deleting execution evidence would
break audit and replay guarantees.

Migration 160 explicitly preflights duplicate legacy `rule_run_id` samples and
legacy samples over 100 rows before adding constraints. Migration 170 adds
leases, exact task deployment identity, bounded replay fields, evidence
digests, and SQL staging receipts. Migration 180 adds the Runner ledger lease,
bounded cleanup claims, retained sample schema fields, and the
`unknown`/`expired` reconciliation states. Migration 190 adds expiring cleanup
claim leases for violation samples and SQL staging receipts, allowing a crashed
cleanup worker to be taken over through compare-and-set after lease expiry.

The local Docker PostgreSQL was upgraded through the formal
`20260723_150 -> head` Alembic path. Two isolated real PostgreSQL tests prove
that duplicate and oversized legacy samples abort with a clear diagnostic and
leave Alembic at revision 150.

## RED / GREEN evidence

The initial fail-first slice produced five expected failures:

- four executor evidence tests failed because `RulePlanExecutor` did not accept
  an evidence writer;
- the Kestra artifact-only test failed because downstream parameters contained
  only workflow inputs and no upstream output reference.

A second fail-first assertion proved that the Polars adapter did not expose a
bounded violation sample before the worker implementation was added.

First review verification:

- focused Runner/rule/Kestra/schema suite: `129 passed`;
- real PostgreSQL/MySQL/MinIO evidence acceptance: `1 passed`;
- real SQL adapter acceptance: `4 passed`;
- real malformed `150 -> head` migration acceptance: `2 passed`;
- full suite: `594 passed, 28 skipped, 59 subtests passed`;
- Ruff across every changed Python file: `All checks passed!`;
- `git diff --check`: passed;
- Docker Compose configuration validation: passed;
- historical migration 140/150 diff check: empty.

## Second-review reconciliation hardening

The second review closes the remaining crash and identity gaps:

- the Runner task ledger has a bounded lease. A crash after claiming a token
  but before creating `rule_runs` atomically converges to `unknown`;
- an existing `rule_runs` row with an expired evidence lease also converges to
  `unknown`, finalizes the exact ledger JTI, and cannot stay at HTTP 202
  permanently;
- expired task tokens are replay-only for a bounded window. Only the exact
  stored JTI, node digest, deployment/environment/workflow binding, and
  response digest may return a prior success. Missing, running, out-of-window,
  or tampered requests are rejected without execution;
- the scheduling gateway now resolves and persists the canonical
  `dataflow_deployments.id`. A workflow candidate ID is never used as a rule
  deployment ID;
- the real gateway path binds the released governance DataFlow version to the
  exact workflow version, schedule, and environment before issuing canary
  tokens;
- rule adapters run with a periodic lease heartbeat. Heartbeat failure records
  `unknown` and preserves a known committed adapter outcome in evidence;
- violation-sample reconciliation claims at most a bounded batch, performs
  MinIO work outside database transactions, verifies digest/schema/count, and
  safely releases retryable unknown claims;
- expired sample deletion is a claim, object deletion, and conditional
  database deletion sequence, so no object network call occurs while holding
  a database transaction;
- SQL staging receipts have equivalent bounded expiry claims and immutable
  audit retention;
- all Kestra predecessor references, including non-rule fan-in, use
  bracket-safe expressions.

Second-review verification:

- targeted rule, Runner, scheduling, Kestra, and schema suite:
  `52 passed`;
- task-token and Runner replay boundary suite: `12 passed`;
- real PostgreSQL/MySQL/MinIO/Gateway/Runner/evidence acceptance:
  `1 passed`;
- full suite: `601 passed, 28 skipped, 59 subtests passed`;
- Ruff across every file changed by the second review:
  `All checks passed!`;
- `git diff --check`: passed;
- migrations 160 and 170 diff against the first-review commit: empty;
- real local PostgreSQL upgrade: `20260723_170 -> 20260723_180 (head)`;
- rebuilt local Docker backend and Runner: both healthy;
- Runner `/health`: `{"status":"ok"}`;
- Alembic inside the rebuilt backend container:
  `20260723_180 (head)`.

The full-repository Ruff invocation still reports 430 pre-existing findings
outside this change set. No changed file contributes a Ruff finding.

## Third-review closed replay and cleanup takeover

The third review makes terminal evidence and cleanup ownership explicit:

- Runner recovery accepts only three exact replay shapes: `running`,
  `missing`, or a fully closed `terminal` response. A terminal success must
  carry a matching canonical result digest, evidence digest, status, and commit
  outcome before the durable ledger may be finalized;
- a genuinely running execution always returns HTTP 202 and never stores a
  success response in the ledger. An expired replay-only token pointing to
  running evidence is rejected with HTTP 401;
- terminal replay responses returned by the PostgreSQL evidence writer include
  both the canonical result digest and the persisted evidence digest, so the
  API does not infer success from an arbitrary dictionary;
- violation-sample and SQL-receipt cleanup claims now have explicit expiry.
  Workers claim rows with compare-and-set semantics, cannot steal an active
  claim, and can take over a claim left behind by a crashed worker after its
  lease expires;
- violation sample reconciliation checks object existence through the storage
  adapter before promotion. A confirmed missing object becomes failed and is
  eligible for cleanup, while transient storage/network errors retain the
  original pending or unknown state and release the claim for retry;
- object-store operations remain outside database transactions, and every
  success, failure, and retry path clears both the claim owner and expiry.

Third-review verification:

- focused Runner, evidence, token, ledger, schema, and real integration suite:
  `41 passed`;
- real PostgreSQL/MySQL/MinIO integration acceptance, including live running
  replay, closed terminal digest recovery, cleanup crash/takeover, confirmed
  missing object, and transient storage failure: `1 passed`;
- full suite: `605 passed, 28 skipped, 59 subtests passed`;
- Ruff across every file changed by the third review:
  `All checks passed!`;
- `git diff --check`: passed;
- migration 180 diff against the second-review commit: empty;
- real local PostgreSQL upgrade: `20260723_180 -> 20260723_190 (head)`;
- rebuilt local Docker backend and Runner: both healthy;
- Runner `/health`: `{"status":"ok"}`;
- Alembic inside the rebuilt backend container:
  `20260723_190 (head)`.

## Fourth-review revision-180 claim compatibility

The fourth review closes the upgrade boundary for cleanup claims created by
revision-180 workers:

- revision 180 already allowed a cleanup worker to persist a non-null
  `cleanup_claim`, but it had no claim-expiry column;
- migration 190 now backfills every such historical sample and SQL-receipt
  claim with `cleanup_claim_expires_at = CURRENT_TIMESTAMP` immediately after
  adding the column;
- rows without a claim remain unclaimed with a null expiry;
- the deployment contract requires revision-180 cleanup workers to be stopped
  during the schema upgrade. This makes immediate expiry safe and prevents an
  old worker from racing a revision-190 takeover;
- after upgrade, the normal owner-and-expiry compare-and-set contract can
  replace the old claim, so a crash before migration cannot permanently lock
  either evidence table.

The real migration acceptance creates an isolated PostgreSQL database, upgrades
it to revision 180, inserts non-null legacy claims into both
`rule_violation_samples` and `rule_sql_staging_receipts`, and then upgrades to
190. It verifies both expiry values are non-null and already reclaimable, then
successfully replaces both old owners through the production-compatible CAS
predicate.

Fourth-review verification:

- focused schema, real migration, Runner, evidence, ledger, and token suite:
  `42 passed`;
- isolated real PostgreSQL `180 -> 190` legacy-claim upgrade:
  passed for both evidence tables;
- full suite: `606 passed, 28 skipped, 59 subtests passed`;
- changed-file Ruff and `git diff --check`: passed;
- migration 180 remains byte-for-byte unchanged from the second-review
  baseline.

## Real PostgreSQL and MinIO acceptance

The production-path integration uses:

- source PostgreSQL and source MySQL;
- platform PostgreSQL at migration head;
- real MinIO;
- the isolated Polars worker;
- `PostgresRulePlanRepository`;
- `PostgresArtifactResolver`;
- `PostgresRuleEvidenceWriter`;
- `RulePlanExecutor`;
- signed Runner HTTP tokens and the durable task ledger.

It proves:

1. the first node reads the PostgreSQL and MySQL artifacts, executes
   normalize/join/assert/deduplicate, publishes an output, and records one
   successful run;
2. one violating row is stored as an all-field-redacted, expiring Parquet
   sample with ready state and exact digest;
3. a second published rule node consumes the first node's exact
   `output_artifact` reference and publishes a distinct output;
4. the same correlation contains exactly two successful run records;
5. a different signed token for the same node replays the existing run and
   sample instead of executing again;
6. same-JTI retry returns the exact stored response without invoking the
   adapter again, and a lost HTTP/ledger response is rebuilt from terminal
   rule evidence;
7. the sample-ready/run-finalize crash window is retried safely without
   changing evidence or leaking raw values;
8. the lease owner is enforced, heartbeat extends only the owned run, and an
   expired lease becomes `unknown` instead of being reclaimed;
9. a real opaque SQL receipt is unavailable while pending, becomes executable
   only after a committed producer finish, and rejects forged, cross-
   correlation, and failed-producer references;
10. same-status/different-content finish retries are rejected by the exact
    evidence digest;
11. real MinIO reconciliation prevents active-claim theft, takes over an
    expired claim, marks confirmed-missing evidence failed, preserves transient
    failures for retry, and removes an unreferenced old object;
12. cleanup claim takeover also applies to opaque SQL staging receipts;
13. a failed adapter records `failed/not_committed`;
14. an uncertain write records `unknown/unknown`;
15. all test-owned PostgreSQL rows and MinIO objects are removed after
   acceptance.

## Residual boundary

Task 6 deliberately does not decide whether a compiled plan may become
published. Task 7 must require trusted generation, test, and preflight evidence
before changing plan lifecycle state. The evidence added here is the immutable
input to that gate, not a bypass around it.