DataOps-platform/.superpowers/sdd/task-6-report.md

Task 6 Report — Artifact handoff and governed rule evidence

Date: 2026-07-23 Branch: codex/data-rule-execution-m3a-m5

Lifecycle boundary: Task 6 consumes only plans already marked published. It does not add, relax, or emulate the Task 7 publication gate.

Outcome

Task 6 now provides a production-wired path from a signed Runner task to one server-attested rule_runs record, optional durable violation evidence, and a closed artifact-only handoff to the next node.

The implementation includes:

• PostgresRuleEvidenceWriter with canonical deployment, component, rule version, plan hash, DataFlow, workflow version, node, and correlation attestation;
• a server-derived evidence key that makes a node execution idempotent across different signed task tokens and Runner retries;
• reliable running to success, failed, unknown, or cancelled finalization with row counts, timings, commit outcome, and bounded public result;
• commit-acknowledgement rechecks for both run and sample finalization;
• a maximum 100-row violation sample, redacted before it leaves the isolated Polars worker and redacted again at the evidence boundary;
• a pending to verified MinIO upload to ready sample state machine with digest, schema hash, reference, expiry, and bounded TTL cleanup;
• no raw sample values in Runner responses or application log calls;
• correlation-scoped upstream output resolution that accepts only current, ready, unexpired, storage-attested artifacts;
• a closed SQL staging reference for same-source pushdown handoff instead of row JSON;
• Kestra single-predecessor input_artifact and multi-predecessor named input_artifacts expressions, with governed rule nodes receiving no unrelated workflow parameters;
• a top-level Runner output_artifact field matching the Kestra expression contract while retaining the full bounded result object.

Forward-only schema change

Migration 20260723_160 extends the existing evidence tables without changing historical migrations 110, 140, or 150.

rule_runs gains:

• nullable legacy-compatible evidence_key with a unique constraint;
• commit_outcome;
• bounded public_result;
• failure_code and updated_at.

rule_violation_samples gains:

• artifact and schema digests;
• legacy, pending, ready, and failed handoff states;
• failure and update fields;
• one sample per run;
• a strict 100-row ceiling and retention index.

Downgrade is deliberately rejected because deleting execution evidence would break audit and replay guarantees.

The local Docker PostgreSQL was upgraded through the formal 20260723_150 -> 20260723_160 Alembic path. The existing isolated old-140 migration acceptance also passes through current head.

RED / GREEN evidence

The initial fail-first slice produced five expected failures:

• four executor evidence tests failed because RulePlanExecutor did not accept an evidence writer;
• the Kestra artifact-only test failed because downstream parameters contained only workflow inputs and no upstream output reference.

A second fail-first assertion proved that the Polars adapter did not expose a bounded violation sample before the worker implementation was added.

Final verification:

• focused evidence, Runner, Polars, Kestra, migration, and real integration: 27 passed;
• full suite: 586 passed, 26 skipped, 59 subtests passed;
• Ruff across every changed Python file: All checks passed!;
• git diff --check: passed;
• Docker Compose configuration validation: passed;
• historical migration 140/150 diff check: empty.

Real PostgreSQL and MinIO acceptance

The production-path integration uses:

• source PostgreSQL and source MySQL;
• platform PostgreSQL at migration head;
• real MinIO;
• the isolated Polars worker;
• PostgresRulePlanRepository;
• PostgresArtifactResolver;
• PostgresRuleEvidenceWriter;
• RulePlanExecutor;
• signed Runner HTTP tokens and the durable task ledger.

It proves:

1. the first node reads the PostgreSQL and MySQL artifacts, executes normalize/join/assert/deduplicate, publishes an output, and records one successful run;
2. one violating row is stored as an all-field-redacted, expiring Parquet sample with ready state and exact digest;
3. a second published rule node consumes the first node's exact output_artifact reference and publishes a distinct output;
4. the same correlation contains exactly two successful run records;
5. a different signed token for the same node replays the existing run and sample instead of executing again;
6. a failed adapter records failed/not_committed;
7. an uncertain write records unknown/unknown;
8. all test-owned PostgreSQL rows and MinIO objects are removed after acceptance.

Residual boundary

Task 6 deliberately does not decide whether a compiled plan may become published. Task 7 must require trusted generation, test, and preflight evidence before changing plan lifecycle state. The evidence added here is the immutable input to that gate, not a bypass around it.