Task 8 Report — Standard / Data Flow product convergence

Date: 2026-07-24 Branch: codex/data-rule-execution-m3a-m5

Outcome

Task 8 removes the user-facing parallel rule implementations from Data Standard and Data Flow authoring.

The governed path is now:

describe a constraint in natural language against fixed input/output schema context;
review AI assumptions, ambiguities, model/prompt provenance and the signed generation receipt;
create an immutable draft RuleVersion;
run server-owned logical compilation and isolated sample testing;
publish only after trusted evidence succeeds;
select a published fixed RuleVersion or StandardVersion from one catalog;
assemble those fixed version IDs into ordered production-line stations;
hand the resulting DataFlowSpec to the release/Data Factory boundary.

Generated source code is never editable in the new path. Legacy code and free-text flow rules remain visible only in explicit migration sections and are excluded from new execution semantics.

Trusted catalog and evidence API

The control plane now exposes:

GET /api/rules/catalog
- closed query, asset_type, limit, and offset parameters;
- rules and standards in one paginated contract;
- human-readable name, stable asset UID, fixed version, owner, status, schema binding/scope, impact count, compiler backend, and latest compile/test evidence;
- only published rules with an exact published logical plan, matching compile evidence, matching test evidence, and a publication audit;
- only published standards whose fixed rule bindings are still published.
GET /api/rules/catalog/assets/<rule|standard>/<version_id>/evidence
- safe generation, logical compilation, dry-run, publication and physical plan stages;
- IDs, hashes, lifecycle states, test kinds, and run IDs only;
- no source text, generated candidate body, artifacts, samples, or unbounded evidence documents.

Compatibility aliases remain available at /api/rules/catalog/rule-versions and /api/rules/rule-versions/<version_id>/evidence. Global API policy keeps all GET paths behind rules:read.

AI authoring lifecycle

RuleAuthoringPanel now:

blocks interpretation until input and output context are present;
displays clarification requirements rather than guessing;
shows assumptions, unresolved ambiguities, confidence, model/provider, prompt revision, repair count, and generation receipt status;
creates a draft with the signed generation receipt;
exposes explicit compile, isolated test and publish actions;
enforces rules:edit and rules:publish UI gates;
displays the trusted evidence timeline;
never renders or edits generated code or a serialized candidate body.

Data Standard convergence

Data Standard editing now links a published RuleVersion through RuleCatalogPicker. New saves contain the fixed rule_version_id and bounded migration_metadata; they do not require or write an operation-code field.

Existing operation code is shown in a read-only migration area with clear unmigrated / linked state and a path back to natural-language re-authoring. It cannot become execution semantics merely by saving the legacy record.

Data Flow production-line assembly

Data Flow no longer edits free-text rules or embeds rule definitions. ProductionLineAssembler models the flow as ordered factory stations:

standard.enforce fixes a standard_version_id;
rule.apply and quality.check fix a rule_version_id;
input/output selections are shown as dataset edges;
schema-compatibility and latest-evidence summaries are shown per station;
missing UID, dataset edges, fixed assets, compatibility, or evidence become explicit release-readiness blockers;
reorder, remove and keyboard-accessible catalog selection are supported.

The new saved script_requirement envelope contains only:

dataflow_spec;
dataset_edges;
bounded migration_metadata.

It contains no inline rule definition or generated source. Existing legacy requirements remain read-only and are not copied into that envelope.

UX and accessibility

The implementation reuses Vue 2, Vuetify, Material Design Icons, and the existing request client. No dependency or design system was added.

The interface uses a low-motion, medium/high-density governance-workbench style. Catalog loading, empty and error states are explicit. Catalog items support focus, arrow navigation, Enter/Space selection, listbox semantics, ARIA labels, and skeleton loading. Responsive layout collapses the Data Flow dataset grid on narrow screens.

Acceptance evidence

Fail-first:

the initial frontend contract run produced 5 failed, 3 passed because the three governed components and convergence behavior did not exist.

Final:

Task 8 API/repository/frontend, Saga and legacy-cutover contracts: 64 passed;
full repository suite: 673 passed, 33 skipped, 59 subtests passed;
git diff --check: passed;
frontend production build: completed with 0 errors;
build retained 20 pre-existing no-console warnings plus existing CSS ordering and bundle-size warnings;
unified catalog SQL executed against the migrated local PostgreSQL service and returned a valid empty page;
rebuilt local Docker backend and frontend images successfully;
backend, frontend, PostgreSQL, Neo4j, MinIO and Runner containers reported healthy;
/api/system/health returned application code 200;
unauthenticated /api/rules/catalog returned HTTP 401.

Browser automation opened the rebuilt deployment and verified the local login surface. The documented local sample administrator password did not match the persisted administrator in this long-lived Docker volume, so authenticated Standard/Data Flow navigation was not forced by overwriting or resetting credentials. The build, component contracts, authenticated API tests, real PostgreSQL query, and container health checks provide the Task 8 acceptance evidence; an authenticated visual walkthrough can be repeated during the phase-level M4 acceptance with the environment owner’s current local credential.

Review closeout

The post-implementation review found eight convergence gaps. All eight are closed:

idempotency is emitted only for rule.apply; standard.enforce and quality.check now match the closed DataFlow contract;
catalog compatibility is never inferred from an arbitrary non-empty object; absent production-line context remains explicitly unknown;
POST /api/interface/data/standard/code is closed with HTTP 410 and read-only-migration semantics;
Standard add/update rejects executable code and server-attests any rule_version_id against the complete trusted publication evidence chain;
DataFlow drafts receive a server-owned UUIDv7 before assembly and saving fails closed if no identity can be obtained;
governed DataFlow create/update validates a closed envelope, verifies all referenced versions are published, preserves the immutable identity, and never invokes task-list, generated-script, n8n-workflow, or legacy data product side effects;
Standard authoring no longer fabricates a client-side validated state; new Standard clauses must bind a server-published RuleVersion;
generation evidence state ready is rendered as successful and async station values are reloaded by the existing deep watcher.

The cutover deliberately preserves legacy non-governed records and read-only migration displays, while preventing those fields from becoming new governed execution semantics.

Security review closeout

The second review strengthened the boundary from server-issued identifiers to database-enforced, single-use draft reservations:

migration 20260723_210 adds a unique DataFlow UID, actor foreign key, unique nonce digest, expiration and consumption ledger;
the draft endpoint persists a short-lived reservation and returns a closed receipt; governed creation atomically consumes it using one conditional UPDATE ... RETURNING;
UID, actor, nonce digest, expiry and unused state must all match, so self-created UUIDs, replay, cross-actor use, expiration and concurrent duplicate consumption fail closed;
an existing governed DataFlow requires the complete governed envelope on every update, keeps the existing UID and cannot be downgraded through script_type, script path, task, code or workflow fields;
any partial governed signal is classified as governed and rejected before legacy task/script/n8n side effects can run.

Data Standard natural-language authoring now deterministically produces a Rule candidate. The prompt states that rule explicitly, the authoring agent repairs a model-produced Standard candidate, and the API independently checks the post-validation candidate before recording or signing it. New legacy Standard records require an exactly attested published RuleVersion; only an existing legacy record may be updated without one for read-only migration.

Catalog compatibility is now evaluated server-side against the production line's current input and output schema references. It returns compatible, incompatible, or unknown with a reason and evidence. The frontend supplies this context, blocks unknown/incompatible stations, hydrates an already-fixed version through an exact-version endpoint even when it is outside the first catalog page, and uses a monotonically increasing request sequence so stale searches cannot overwrite newer context.

Real PostgreSQL acceptance confirmed:

Alembic upgraded the persisted Docker database from revision 200 to 20260723_210 (head);
the reservation table exists and its actor foreign key rejects an unknown actor;
two independent concurrent database sessions consuming one receipt produce exactly one acceptance and one rejection;
replay, cross-actor and expired receipt consumption are rejected;
acceptance fixtures are removed after the test.

Recoverable create and trusted-schema closeout

The third review replaced the one-shot PostgreSQL-consume/Neo4j-create boundary with a recoverable create Saga. Historical migration 210 remains unchanged; forward-only migration 20260724_220 adds:

reserved, creating, completed, and failed states;
an expiring create lease, attempt counter and error code;
the Neo4j node ID, bounded result JSON and canonical SHA-256 digest;
create-started, failed, completed and updated timestamps.

begin_dataflow_create now uses one conditional UPDATE ... RETURNING to claim a lease. The claim is committed before the graph side effect. Governed DataFlow creation installs the idempotent DataFlow.uid unique constraint, uses MERGE only on the server-reserved UID, and verifies the persisted name, UID, script type and canonical governed requirement without overwriting an existing node. Completion conditionally owns the lease and persists the exact response. A retry after response loss returns that stored response; an expired lease reclaims the same UID, observes the existing graph node and finalizes it. Name/UID or immutable-property conflicts fail closed.

Catalog compatibility no longer trusts matching reference strings alone:

the current input/output references are resolved through SchemaResolver;
published rules compare the current snapshot hashes with the hashes frozen into their published logical plan;
compile and test evidence IDs must be tied to that same logical plan and hash set;
a Standard expands every fixed standard_rule_bindings entry and requires the same trusted rule plan/evidence/hash chain for every clause;
resolver failure returns unknown; same-reference hash drift and any bound Standard rule drift return incompatible;
reader evidence contains only stable IDs, hashes, snapshot IDs and source revisions.

The Data Flow editor stores the reservation expiry, renews it when less than 60 seconds remain, and changes to a new server UID only when the server returns the explicit draft_reservation_refresh_required signal. It retries that safe pre-side-effect case once. Network errors and in-progress leases do not cause a new UID; a completed response-loss retry is replayed by the server.

Additional acceptance evidence:

local PostgreSQL upgraded from 210 to 20260724_220 (head);
two real concurrent PostgreSQL sessions produced one claim and one dataflow_create_in_progress; completion replay returned the identical result, expired drafts were rejected, and an expired create lease was reclaimed as attempt 2;
real Neo4j installed the data_flow_uid unique constraint, two same-UID merges produced one node, and immutable drift was rejected;
fault-injection tests cover Neo4j failure after a durable PG claim, PostgreSQL finalize failure after Neo4j success, completed-response replay, conflict closure and lease recovery;
real PostgreSQL and Neo4j integration: 2 passed;
rebuilt backend and frontend images report migration 230, healthy database and Neo4j checks, application health code 200, and frontend HTTP 200.

Durable snapshot and orphan-reconciliation closeout

The fourth review closed the remaining transaction and recovery boundaries:

catalog endpoints now treat SchemaResolver as an explicit read-through persistence boundary: successful context-aware GETs commit newly created snapshots before returning their IDs, and every error path rolls back;
real PostgreSQL/API acceptance issued two separate catalog requests, observed one stable snapshot ID, and queried that ID from an independent database session;
forward-only migration 20260724_230 adds the canonical create request, server-owned create intent and SHA-256 request digest without modifying migration 220;
first claim atomically freezes request, intent and digest; every later claim, completed replay and finalize must match that digest;
changing the name or any other canonical payload/intent value while reusing a receipt fails with dataflow_create_request_conflict instead of silently returning an older result;
governed creation uses the supplied English name or a deterministic UID-derived fallback and never calls non-deterministic translation before computing the request digest.

DataFlowCreateReconciler and python -m app.commands.reconcile_dataflow_creates recover orphaned graph nodes without accepting a nonce or inline business payload:

dry-run is the default and reports bounded safe IDs, state, digest and attempt information without changing state or attempt count;
apply mode claims expired creating or retryable failed rows with FOR UPDATE SKIP LOCKED and an expiring lease;
canonical request digest, duplicated intent and DataFlow UID are verified before any Neo4j write;
the existing UID-only graph MERGE either finds the orphan or creates the missing node, verifies immutable properties, and then finalizes PostgreSQL;
corrupt intent rows receive a terminal integrity error and do not block valid rows; other failures are isolated per reservation and remain retryable;
two real concurrent reconcilers produced exactly one claim;
real PostgreSQL/Neo4j acceptance created a graph node, deliberately omitted PostgreSQL finalize, discarded the receipt, expired the lease, and then completed the Saga from stored intent with exactly one graph node;
the reconciliation CLI dry-run completed against the local database with an empty, non-mutating report.

Fourth-review acceptance:

focused repository/API/Saga/cutover contracts: 65 passed;
real PostgreSQL/API/Neo4j integration: 4 passed;
full repository suite: 673 passed, 33 skipped, 59 subtests passed.

Fenced reconciliation and strict-intent closeout

The fifth review removed the remaining concurrency and poison-record failure modes from orphan reconciliation:

apply mode is bounded by the requested total but claims only one row at a time, commits that claim, processes it and only then claims the next row; a slow first item therefore cannot leave a batch of later items waiting on leases that expire before their processing begins;
after the trusted intent is semantically validated and immediately before the Neo4j side effect, the reconciler conditionally renews the lease using the reservation ID and current lease token. Lost ownership prevents the side effect and is reported as lease_recovery_required;
reconciliation parsing is isolated per claimed row. Scalar, malformed-shape or oversized request/intent JSON becomes terminal create_intent_integrity_failed without aborting the next valid item;
semantic intent validation failures become terminal create_intent_validation_failed. Both terminal codes are excluded from later list and claim queries, while infrastructure failures remain retryable;
failure persistence uses UPDATE ... RETURNING; a stale worker that no longer owns the lease cannot report a false successful failure transition;
governed tag relationships now use one atomic Neo4j MERGE (DataFlow)-[:LABEL]->(DataLabel) path shared by normal creation and reconciliation. Governed tag failures are strict: PostgreSQL is not finalized, the Saga remains retryable, and a later retry completes without duplicate relationships. Legacy tag handling retains its previous best-effort behavior.

Fifth-review acceptance:

focused Saga/repository/cutover contracts: 40 passed;
real PostgreSQL/API/Neo4j integration: 3 passed;
a full repository run exposed one overly narrow receipt-tamper assertion: modified base64url input can correctly fail either canonical encoding or signature verification. The assertion now accepts both fail-closed reasons; the fresh final run is 681 passed, 33 skipped, 59 subtests passed;
selected Ruff F, I, and B checks passed;
the production frontend build completed with zero errors and the same existing 20 console warnings;
the backend image was rebuilt from the reviewed source; backend and frontend containers are healthy, Alembic reports 20260724_230 (head), application health reports database and Neo4j healthy with code 200, and frontend HTTP returns 200.

DataFlow name concurrency closeout

The sixth review closed the final governed-create time-of-check/time-of-use window around DataFlow.name_zh:

governed graph creation now installs and consumes both data_flow_uid and data_flow_name_zh uniqueness constraints before its preflight and UID-keyed MERGE;
constraint installation scans existing graph state. Historical duplicate UIDs or Chinese names make installation fail closed; they are never silently accepted or rewritten;
concurrent first-time schema installation retries only Neo4j deadlock victims. Constraint validation/verification/creation failures and concurrent unique-property violations are normalized to dataflow_uid_conflict;
the existing same-name preflight remains useful diagnostics, while the database uniqueness constraint is authoritative at the concurrent write boundary.

Sixth-review acceptance:

focused Saga/repository/publication/cutover contracts: 47 passed;
the dedicated real-Neo4j integration suite is 2 passed: two concurrent creates with different UUIDv7 values and the same name_zh produced exactly one success and one dataflow_uid_conflict, left one node, and verified both constraints;
a simulated historical duplicate made name-constraint installation fail before any graph MERGE;
full repository suite: 682 passed, 34 skipped, 59 subtests passed;
selected Ruff F, I, and B checks and whitespace validation passed;
the production frontend build completed with zero errors and the same existing 20 console warnings;
the final backend image was rebuilt from the reviewed source; backend and frontend are healthy, Alembic remains 20260724_230 (head), application health reports database and Neo4j healthy with code 200, and frontend HTTP returns 200.

Residual scope

Production-line cross-station compatibility remains ultimately authoritative at server-side release resolution; the UI presents current bound schema evidence and blocks on missing evidence rather than claiming runtime success.
Data Factory activation, canary, rollback, deployment permissions and environment operations remain Task 9 scope.
The repository has no published catalog fixtures in the current persisted local database, so catalog empty state was the live deployment state during this acceptance.

task-8-report.md 20 KB History Raw