# Task 6 Report — Artifact handoff and governed rule evidence

Date: 2026-07-23
Branch: `codex/data-rule-execution-m3a-m5`

Lifecycle boundary: Task 6 consumes only plans already marked `published`.
It does not add, relax, or emulate the Task 7 publication gate.

## Outcome

Task 6 now provides a production-wired path from a signed Runner task to one
server-attested `rule_runs` record, optional durable violation evidence, and a
closed artifact-only handoff to the next node.

The implementation includes:

- `PostgresRuleEvidenceWriter` with canonical deployment, component, rule
  version, plan hash, DataFlow, workflow version, node, and correlation
  attestation;
- a server-derived evidence key that makes a node execution idempotent across
  different signed task tokens and Runner retries;
- reliable `running` to `success`, `failed`, `unknown`, or `cancelled`
  finalization with row counts, timings, commit outcome, and bounded public
  result;
- a signed, exact deployment/environment task identity plus a durable lease
  owner, heartbeat, and expiry boundary; an expired run is finalized
  `unknown` and is never blindly re-executed;
- same-JTI response replay from the durable ledger, including recovery from
  terminal rule evidence when execution committed but the HTTP/ledger response
  was lost;
- commit-acknowledgement rechecks for both run and sample finalization;
- a maximum 100-row violation sample, redacted before it leaves the isolated
  Polars worker and redacted again at the evidence boundary;
- a `pending` to verified MinIO upload to `ready` sample state machine with
  digest, schema hash, reference, expiry, and bounded TTL cleanup;
- no raw sample values in Runner responses or application log calls;
- correlation-scoped upstream output resolution that accepts only current,
  ready, unexpired, storage-attested artifacts;
- an opaque `dataops-staging://<uuid>` SQL receipt for same-source pushdown
  handoff, bound to producer run, deployment, correlation, dataset binding,
  binding hash, relation digest, commit state, and expiry;
- bracket-safe Kestra single-predecessor `input_artifact` expressions;
  governed rule fan-in is rejected until an explicit merge component exists;
- a top-level Runner `output_artifact` field matching the Kestra expression
  contract while retaining the full bounded result object.

## Forward-only schema change

Migrations `20260723_160` and `20260723_170` extend the existing evidence
tables without changing historical migrations 110, 140, or 150.

`rule_runs` gains:

- nullable legacy-compatible `evidence_key` with a unique constraint;
- `commit_outcome`;
- bounded `public_result`;
- `failure_code` and `updated_at`.

`rule_violation_samples` gains:

- artifact and schema digests;
- `legacy`, `pending`, `ready`, and `failed` handoff states;
- failure and update fields;
- one sample per run;
- a strict 100-row ceiling and retention index.

Downgrade is deliberately rejected because deleting execution evidence would
break audit and replay guarantees.

Migration 160 explicitly preflights duplicate legacy `rule_run_id` samples and
legacy samples over 100 rows before adding constraints. Migration 170 adds
leases, exact task deployment identity, bounded replay fields, evidence
digests, and SQL staging receipts.

The local Docker PostgreSQL was upgraded through the formal
`20260723_150 -> head` Alembic path. Two isolated real PostgreSQL tests prove
that duplicate and oversized legacy samples abort with a clear diagnostic and
leave Alembic at revision 150.

## RED / GREEN evidence

The initial fail-first slice produced five expected failures:

- four executor evidence tests failed because `RulePlanExecutor` did not accept
  an evidence writer;
- the Kestra artifact-only test failed because downstream parameters contained
  only workflow inputs and no upstream output reference.

A second fail-first assertion proved that the Polars adapter did not expose a
bounded violation sample before the worker implementation was added.

Final review verification:

- focused Runner/rule/Kestra/schema suite: `129 passed`;
- real PostgreSQL/MySQL/MinIO evidence acceptance: `1 passed`;
- real SQL adapter acceptance: `4 passed`;
- real malformed `150 -> head` migration acceptance: `2 passed`;
- full suite: `594 passed, 28 skipped, 59 subtests passed`;
- Ruff across every changed Python file: `All checks passed!`;
- `git diff --check`: passed;
- Docker Compose configuration validation: passed;
- historical migration 140/150 diff check: empty.

## Real PostgreSQL and MinIO acceptance

The production-path integration uses:

- source PostgreSQL and source MySQL;
- platform PostgreSQL at migration head;
- real MinIO;
- the isolated Polars worker;
- `PostgresRulePlanRepository`;
- `PostgresArtifactResolver`;
- `PostgresRuleEvidenceWriter`;
- `RulePlanExecutor`;
- signed Runner HTTP tokens and the durable task ledger.

It proves:

1. the first node reads the PostgreSQL and MySQL artifacts, executes
   normalize/join/assert/deduplicate, publishes an output, and records one
   successful run;
2. one violating row is stored as an all-field-redacted, expiring Parquet
   sample with ready state and exact digest;
3. a second published rule node consumes the first node's exact
   `output_artifact` reference and publishes a distinct output;
4. the same correlation contains exactly two successful run records;
5. a different signed token for the same node replays the existing run and
   sample instead of executing again;
6. same-JTI retry returns the exact stored response without invoking the
   adapter again, and a lost HTTP/ledger response is rebuilt from terminal
   rule evidence;
7. the sample-ready/run-finalize crash window is retried safely without
   changing evidence or leaking raw values;
8. the lease owner is enforced, heartbeat extends only the owned run, and an
   expired lease becomes `unknown` instead of being reclaimed;
9. a real opaque SQL receipt is unavailable while pending, becomes executable
   only after a committed producer finish, and rejects forged, cross-
   correlation, and failed-producer references;
10. same-status/different-content finish retries are rejected by the exact
    evidence digest;
11. real MinIO reconciliation retains pending/ready violation evidence while
    removing an unreferenced old object;
12. a failed adapter records `failed/not_committed`;
13. an uncertain write records `unknown/unknown`;
14. all test-owned PostgreSQL rows and MinIO objects are removed after
   acceptance.

## Residual boundary

Task 6 deliberately does not decide whether a compiled plan may become
published. Task 7 must require trusted generation, test, and preflight evidence
before changing plan lifecycle state. The evidence added here is the immutable
input to that gate, not a bypass around it.