Task 6 Report — Artifact handoff and governed rule evidence

Date: 2026-07-23 Branch: codex/data-rule-execution-m3a-m5

Lifecycle boundary: Task 6 consumes only plans already marked published. It does not add, relax, or emulate the Task 7 publication gate.

Outcome

Task 6 now provides a production-wired path from a signed Runner task to one server-attested rule_runs record, optional durable violation evidence, and a closed artifact-only handoff to the next node.

The implementation includes:

PostgresRuleEvidenceWriter with canonical deployment, component, rule version, plan hash, DataFlow, workflow version, node, and correlation attestation;
a server-derived evidence key that makes a node execution idempotent across different signed task tokens and Runner retries;
reliable running to success, failed, unknown, or cancelled finalization with row counts, timings, commit outcome, and bounded public result;
a signed, exact deployment/environment task identity plus a durable lease owner, heartbeat, and expiry boundary; an expired run is finalized unknown and is never blindly re-executed;
same-JTI response replay from the durable ledger, including recovery from terminal rule evidence when execution committed but the HTTP/ledger response was lost;
commit-acknowledgement rechecks for both run and sample finalization;
a maximum 100-row violation sample, redacted before it leaves the isolated Polars worker and redacted again at the evidence boundary;
a pending to verified MinIO upload to ready sample state machine with digest, schema hash, reference, expiry, and bounded TTL cleanup;
no raw sample values in Runner responses or application log calls;
correlation-scoped upstream output resolution that accepts only current, ready, unexpired, storage-attested artifacts;
an opaque dataops-staging://<uuid> SQL receipt for same-source pushdown handoff, bound to producer run, deployment, correlation, dataset binding, binding hash, relation digest, commit state, and expiry;
bracket-safe Kestra single-predecessor input_artifact expressions; governed rule fan-in is rejected until an explicit merge component exists;
a top-level Runner output_artifact field matching the Kestra expression contract while retaining the full bounded result object.

Forward-only schema change

Migrations 20260723_160, 20260723_170, 20260723_180, and 20260723_190 extend the existing evidence tables without changing historical migrations 110, 140, or 150. The second review did not rewrite already-applied migrations 160 or 170, and the third review leaves migration 180 unchanged. Each additional schema revision is isolated in its own forward-only migration.

rule_runs gains:

nullable legacy-compatible evidence_key with a unique constraint;
commit_outcome;
bounded public_result;
failure_code and updated_at.

rule_violation_samples gains:

artifact and schema digests;
legacy, pending, ready, and failed handoff states;
failure and update fields;
one sample per run;
a strict 100-row ceiling and retention index.

Downgrade is deliberately rejected because deleting execution evidence would break audit and replay guarantees.

Migration 160 explicitly preflights duplicate legacy rule_run_id samples and legacy samples over 100 rows before adding constraints. Migration 170 adds leases, exact task deployment identity, bounded replay fields, evidence digests, and SQL staging receipts. Migration 180 adds the Runner ledger lease, bounded cleanup claims, retained sample schema fields, and the unknown/expired reconciliation states. Migration 190 adds expiring cleanup claim leases for violation samples and SQL staging receipts, allowing a crashed cleanup worker to be taken over through compare-and-set after lease expiry.

The local Docker PostgreSQL was upgraded through the formal 20260723_150 -> head Alembic path. Two isolated real PostgreSQL tests prove that duplicate and oversized legacy samples abort with a clear diagnostic and leave Alembic at revision 150.

RED / GREEN evidence

The initial fail-first slice produced five expected failures:

four executor evidence tests failed because RulePlanExecutor did not accept an evidence writer;
the Kestra artifact-only test failed because downstream parameters contained only workflow inputs and no upstream output reference.

A second fail-first assertion proved that the Polars adapter did not expose a bounded violation sample before the worker implementation was added.

First review verification:

focused Runner/rule/Kestra/schema suite: 129 passed;
real PostgreSQL/MySQL/MinIO evidence acceptance: 1 passed;
real SQL adapter acceptance: 4 passed;
real malformed 150 -> head migration acceptance: 2 passed;
full suite: 594 passed, 28 skipped, 59 subtests passed;
Ruff across every changed Python file: All checks passed!;
git diff --check: passed;
Docker Compose configuration validation: passed;
historical migration 140/150 diff check: empty.

Second-review reconciliation hardening

The second review closes the remaining crash and identity gaps:

the Runner task ledger has a bounded lease. A crash after claiming a token but before creating rule_runs atomically converges to unknown;
an existing rule_runs row with an expired evidence lease also converges to unknown, finalizes the exact ledger JTI, and cannot stay at HTTP 202 permanently;
expired task tokens are replay-only for a bounded window. Only the exact stored JTI, node digest, deployment/environment/workflow binding, and response digest may return a prior success. Missing, running, out-of-window, or tampered requests are rejected without execution;
the scheduling gateway now resolves and persists the canonical dataflow_deployments.id. A workflow candidate ID is never used as a rule deployment ID;
the real gateway path binds the released governance DataFlow version to the exact workflow version, schedule, and environment before issuing canary tokens;
rule adapters run with a periodic lease heartbeat. Heartbeat failure records unknown and preserves a known committed adapter outcome in evidence;
violation-sample reconciliation claims at most a bounded batch, performs MinIO work outside database transactions, verifies digest/schema/count, and safely releases retryable unknown claims;
expired sample deletion is a claim, object deletion, and conditional database deletion sequence, so no object network call occurs while holding a database transaction;
SQL staging receipts have equivalent bounded expiry claims and immutable audit retention;
all Kestra predecessor references, including non-rule fan-in, use bracket-safe expressions.

Second-review verification:

targeted rule, Runner, scheduling, Kestra, and schema suite: 52 passed;
task-token and Runner replay boundary suite: 12 passed;
real PostgreSQL/MySQL/MinIO/Gateway/Runner/evidence acceptance: 1 passed;
full suite: 601 passed, 28 skipped, 59 subtests passed;
Ruff across every file changed by the second review: All checks passed!;
git diff --check: passed;
migrations 160 and 170 diff against the first-review commit: empty;
real local PostgreSQL upgrade: 20260723_170 -> 20260723_180 (head);
rebuilt local Docker backend and Runner: both healthy;
Runner /health: {"status":"ok"};
Alembic inside the rebuilt backend container: 20260723_180 (head).

The full-repository Ruff invocation still reports 430 pre-existing findings outside this change set. No changed file contributes a Ruff finding.

Third-review closed replay and cleanup takeover

The third review makes terminal evidence and cleanup ownership explicit:

Runner recovery accepts only three exact replay shapes: running, missing, or a fully closed terminal response. A terminal success must carry a matching canonical result digest, evidence digest, status, and commit outcome before the durable ledger may be finalized;
a genuinely running execution always returns HTTP 202 and never stores a success response in the ledger. An expired replay-only token pointing to running evidence is rejected with HTTP 401;
terminal replay responses returned by the PostgreSQL evidence writer include both the canonical result digest and the persisted evidence digest, so the API does not infer success from an arbitrary dictionary;
violation-sample and SQL-receipt cleanup claims now have explicit expiry. Workers claim rows with compare-and-set semantics, cannot steal an active claim, and can take over a claim left behind by a crashed worker after its lease expires;
violation sample reconciliation checks object existence through the storage adapter before promotion. A confirmed missing object becomes failed and is eligible for cleanup, while transient storage/network errors retain the original pending or unknown state and release the claim for retry;
object-store operations remain outside database transactions, and every success, failure, and retry path clears both the claim owner and expiry.

Third-review verification:

focused Runner, evidence, token, ledger, schema, and real integration suite: 41 passed;
real PostgreSQL/MySQL/MinIO integration acceptance, including live running replay, closed terminal digest recovery, cleanup crash/takeover, confirmed missing object, and transient storage failure: 1 passed;
full suite: 605 passed, 28 skipped, 59 subtests passed;
Ruff across every file changed by the third review: All checks passed!;
git diff --check: passed;
migration 180 diff against the second-review commit: empty;
real local PostgreSQL upgrade: 20260723_180 -> 20260723_190 (head);
rebuilt local Docker backend and Runner: both healthy;
Runner /health: {"status":"ok"};
Alembic inside the rebuilt backend container: 20260723_190 (head).

Fourth-review revision-180 claim compatibility

The fourth review closes the upgrade boundary for cleanup claims created by revision-180 workers:

revision 180 already allowed a cleanup worker to persist a non-null cleanup_claim, but it had no claim-expiry column;
migration 190 now backfills every such historical sample and SQL-receipt claim with cleanup_claim_expires_at = CURRENT_TIMESTAMP immediately after adding the column;
rows without a claim remain unclaimed with a null expiry;
the deployment contract requires revision-180 cleanup workers to be stopped during the schema upgrade. This makes immediate expiry safe and prevents an old worker from racing a revision-190 takeover;
after upgrade, the normal owner-and-expiry compare-and-set contract can replace the old claim, so a crash before migration cannot permanently lock either evidence table.

The real migration acceptance creates an isolated PostgreSQL database, upgrades it to revision 180, inserts non-null legacy claims into both rule_violation_samples and rule_sql_staging_receipts, and then upgrades to

It verifies both expiry values are non-null and already reclaimable, then successfully replaces both old owners through the production-compatible CAS predicate.

Fourth-review verification:

focused schema, real migration, Runner, evidence, ledger, and token suite: 42 passed;
isolated real PostgreSQL 180 -> 190 legacy-claim upgrade: passed for both evidence tables;
full suite: 606 passed, 28 skipped, 59 subtests passed;
changed-file Ruff and git diff --check: passed;
migration 180 remains byte-for-byte unchanged from the second-review baseline.

Real PostgreSQL and MinIO acceptance

The production-path integration uses:

source PostgreSQL and source MySQL;
platform PostgreSQL at migration head;
real MinIO;
the isolated Polars worker;
PostgresRulePlanRepository;
PostgresArtifactResolver;
PostgresRuleEvidenceWriter;
RulePlanExecutor;
signed Runner HTTP tokens and the durable task ledger.

It proves:

the first node reads the PostgreSQL and MySQL artifacts, executes normalize/join/assert/deduplicate, publishes an output, and records one successful run;
one violating row is stored as an all-field-redacted, expiring Parquet sample with ready state and exact digest;
a second published rule node consumes the first node's exact output_artifact reference and publishes a distinct output;
the same correlation contains exactly two successful run records;
a different signed token for the same node replays the existing run and sample instead of executing again;
same-JTI retry returns the exact stored response without invoking the adapter again, and a lost HTTP/ledger response is rebuilt from terminal rule evidence;
the sample-ready/run-finalize crash window is retried safely without changing evidence or leaking raw values;
the lease owner is enforced, heartbeat extends only the owned run, and an expired lease becomes unknown instead of being reclaimed;
a real opaque SQL receipt is unavailable while pending, becomes executable only after a committed producer finish, and rejects forged, cross- correlation, and failed-producer references;
same-status/different-content finish retries are rejected by the exact evidence digest;
real MinIO reconciliation prevents active-claim theft, takes over an expired claim, marks confirmed-missing evidence failed, preserves transient failures for retry, and removes an unreferenced old object;
cleanup claim takeover also applies to opaque SQL staging receipts;
a failed adapter records failed/not_committed;
an uncertain write records unknown/unknown;
all test-owned PostgreSQL rows and MinIO objects are removed after acceptance.

Residual boundary

Task 6 deliberately does not decide whether a compiled plan may become published. Task 7 must require trusted generation, test, and preflight evidence before changing plan lifecycle state. The evidence added here is the immutable input to that gate, not a bypass around it.

task-6-report.md 14 KB Geçmiş Ham