DataOps-platform/.superpowers/sdd/task-6-report.md

Task 6 Report — Artifact handoff and governed rule evidence

Date: 2026-07-23 Branch: codex/data-rule-execution-m3a-m5

Lifecycle boundary: Task 6 consumes only plans already marked published. It does not add, relax, or emulate the Task 7 publication gate.

Outcome

Task 6 now provides a production-wired path from a signed Runner task to one server-attested rule_runs record, optional durable violation evidence, and a closed artifact-only handoff to the next node.

The implementation includes:

• PostgresRuleEvidenceWriter with canonical deployment, component, rule version, plan hash, DataFlow, workflow version, node, and correlation attestation;
• a server-derived evidence key that makes a node execution idempotent across different signed task tokens and Runner retries;
• reliable running to success, failed, unknown, or cancelled finalization with row counts, timings, commit outcome, and bounded public result;
• a signed, exact deployment/environment task identity plus a durable lease owner, heartbeat, and expiry boundary; an expired run is finalized unknown and is never blindly re-executed;
• same-JTI response replay from the durable ledger, including recovery from terminal rule evidence when execution committed but the HTTP/ledger response was lost;
• commit-acknowledgement rechecks for both run and sample finalization;
• a maximum 100-row violation sample, redacted before it leaves the isolated Polars worker and redacted again at the evidence boundary;
• a pending to verified MinIO upload to ready sample state machine with digest, schema hash, reference, expiry, and bounded TTL cleanup;
• no raw sample values in Runner responses or application log calls;
• correlation-scoped upstream output resolution that accepts only current, ready, unexpired, storage-attested artifacts;
• an opaque dataops-staging://<uuid> SQL receipt for same-source pushdown handoff, bound to producer run, deployment, correlation, dataset binding, binding hash, relation digest, commit state, and expiry;
• bracket-safe Kestra single-predecessor input_artifact expressions; governed rule fan-in is rejected until an explicit merge component exists;
• a top-level Runner output_artifact field matching the Kestra expression contract while retaining the full bounded result object.

Forward-only schema change

Migrations 20260723_160 and 20260723_170 extend the existing evidence tables without changing historical migrations 110, 140, or 150.

rule_runs gains:

• nullable legacy-compatible evidence_key with a unique constraint;
• commit_outcome;
• bounded public_result;
• failure_code and updated_at.

rule_violation_samples gains:

• artifact and schema digests;
• legacy, pending, ready, and failed handoff states;
• failure and update fields;
• one sample per run;
• a strict 100-row ceiling and retention index.

Downgrade is deliberately rejected because deleting execution evidence would break audit and replay guarantees.

Migration 160 explicitly preflights duplicate legacy rule_run_id samples and legacy samples over 100 rows before adding constraints. Migration 170 adds leases, exact task deployment identity, bounded replay fields, evidence digests, and SQL staging receipts.

The local Docker PostgreSQL was upgraded through the formal 20260723_150 -> head Alembic path. Two isolated real PostgreSQL tests prove that duplicate and oversized legacy samples abort with a clear diagnostic and leave Alembic at revision 150.

RED / GREEN evidence

The initial fail-first slice produced five expected failures:

• four executor evidence tests failed because RulePlanExecutor did not accept an evidence writer;
• the Kestra artifact-only test failed because downstream parameters contained only workflow inputs and no upstream output reference.

A second fail-first assertion proved that the Polars adapter did not expose a bounded violation sample before the worker implementation was added.

Final review verification:

• focused Runner/rule/Kestra/schema suite: 129 passed;
• real PostgreSQL/MySQL/MinIO evidence acceptance: 1 passed;
• real SQL adapter acceptance: 4 passed;
• real malformed 150 -> head migration acceptance: 2 passed;
• full suite: 594 passed, 28 skipped, 59 subtests passed;
• Ruff across every changed Python file: All checks passed!;
• git diff --check: passed;
• Docker Compose configuration validation: passed;
• historical migration 140/150 diff check: empty.

Real PostgreSQL and MinIO acceptance

The production-path integration uses:

• source PostgreSQL and source MySQL;
• platform PostgreSQL at migration head;
• real MinIO;
• the isolated Polars worker;
• PostgresRulePlanRepository;
• PostgresArtifactResolver;
• PostgresRuleEvidenceWriter;
• RulePlanExecutor;
• signed Runner HTTP tokens and the durable task ledger.

It proves:

1. the first node reads the PostgreSQL and MySQL artifacts, executes normalize/join/assert/deduplicate, publishes an output, and records one successful run;
2. one violating row is stored as an all-field-redacted, expiring Parquet sample with ready state and exact digest;
3. a second published rule node consumes the first node's exact output_artifact reference and publishes a distinct output;
4. the same correlation contains exactly two successful run records;
5. a different signed token for the same node replays the existing run and sample instead of executing again;
6. same-JTI retry returns the exact stored response without invoking the adapter again, and a lost HTTP/ledger response is rebuilt from terminal rule evidence;
7. the sample-ready/run-finalize crash window is retried safely without changing evidence or leaking raw values;
8. the lease owner is enforced, heartbeat extends only the owned run, and an expired lease becomes unknown instead of being reclaimed;
9. a real opaque SQL receipt is unavailable while pending, becomes executable only after a committed producer finish, and rejects forged, cross- correlation, and failed-producer references;
10. same-status/different-content finish retries are rejected by the exact evidence digest;
11. real MinIO reconciliation retains pending/ready violation evidence while removing an unreferenced old object;
12. a failed adapter records failed/not_committed;
13. an uncertain write records unknown/unknown;
14. all test-owned PostgreSQL rows and MinIO objects are removed after acceptance.

Residual boundary

Task 6 deliberately does not decide whether a compiled plan may become published. Task 7 must require trusted generation, test, and preflight evidence before changing plan lifecycle state. The evidence added here is the immutable input to that gate, not a bypass around it.