# Task 8 Report — Standard / Data Flow product convergence Date: 2026-07-24 Branch: `codex/data-rule-execution-m3a-m5` ## Outcome Task 8 removes the user-facing parallel rule implementations from Data Standard and Data Flow authoring. The governed path is now: 1. describe a constraint in natural language against fixed input/output schema context; 2. review AI assumptions, ambiguities, model/prompt provenance and the signed generation receipt; 3. create an immutable draft RuleVersion; 4. run server-owned logical compilation and isolated sample testing; 5. publish only after trusted evidence succeeds; 6. select a published fixed RuleVersion or StandardVersion from one catalog; 7. assemble those fixed version IDs into ordered production-line stations; 8. hand the resulting DataFlowSpec to the release/Data Factory boundary. Generated source code is never editable in the new path. Legacy code and free-text flow rules remain visible only in explicit migration sections and are excluded from new execution semantics. ## Trusted catalog and evidence API The control plane now exposes: - `GET /api/rules/catalog` - closed `query`, `asset_type`, `limit`, and `offset` parameters; - rules and standards in one paginated contract; - human-readable name, stable asset UID, fixed version, owner, status, schema binding/scope, impact count, compiler backend, and latest compile/test evidence; - only published rules with an exact published logical plan, matching compile evidence, matching test evidence, and a publication audit; - only published standards whose fixed rule bindings are still published. - `GET /api/rules/catalog/assets///evidence` - safe generation, logical compilation, dry-run, publication and physical plan stages; - IDs, hashes, lifecycle states, test kinds, and run IDs only; - no source text, generated candidate body, artifacts, samples, or unbounded evidence documents. Compatibility aliases remain available at `/api/rules/catalog/rule-versions` and `/api/rules/rule-versions//evidence`. Global API policy keeps all GET paths behind `rules:read`. ## AI authoring lifecycle `RuleAuthoringPanel` now: - blocks interpretation until input and output context are present; - displays clarification requirements rather than guessing; - shows assumptions, unresolved ambiguities, confidence, model/provider, prompt revision, repair count, and generation receipt status; - creates a draft with the signed generation receipt; - exposes explicit compile, isolated test and publish actions; - enforces `rules:edit` and `rules:publish` UI gates; - displays the trusted evidence timeline; - never renders or edits generated code or a serialized candidate body. ## Data Standard convergence Data Standard editing now links a published RuleVersion through `RuleCatalogPicker`. New saves contain the fixed `rule_version_id` and bounded `migration_metadata`; they do not require or write an operation-code field. Existing operation code is shown in a read-only migration area with clear `unmigrated` / `linked` state and a path back to natural-language re-authoring. It cannot become execution semantics merely by saving the legacy record. ## Data Flow production-line assembly Data Flow no longer edits free-text rules or embeds rule definitions. `ProductionLineAssembler` models the flow as ordered factory stations: - `standard.enforce` fixes a `standard_version_id`; - `rule.apply` and `quality.check` fix a `rule_version_id`; - input/output selections are shown as dataset edges; - schema-compatibility and latest-evidence summaries are shown per station; - missing UID, dataset edges, fixed assets, compatibility, or evidence become explicit release-readiness blockers; - reorder, remove and keyboard-accessible catalog selection are supported. The new saved `script_requirement` envelope contains only: - `dataflow_spec`; - `dataset_edges`; - bounded `migration_metadata`. It contains no inline rule definition or generated source. Existing legacy requirements remain read-only and are not copied into that envelope. ## UX and accessibility The implementation reuses Vue 2, Vuetify, Material Design Icons, and the existing request client. No dependency or design system was added. The interface uses a low-motion, medium/high-density governance-workbench style. Catalog loading, empty and error states are explicit. Catalog items support focus, arrow navigation, Enter/Space selection, listbox semantics, ARIA labels, and skeleton loading. Responsive layout collapses the Data Flow dataset grid on narrow screens. ## Acceptance evidence Fail-first: - the initial frontend contract run produced `5 failed, 3 passed` because the three governed components and convergence behavior did not exist. Final: - Task 8 API/repository/frontend, Saga and legacy-cutover contracts: `64 passed`; - full repository suite: `673 passed, 33 skipped, 59 subtests passed`; - `git diff --check`: passed; - frontend production build: completed with `0 errors`; - build retained 20 pre-existing `no-console` warnings plus existing CSS ordering and bundle-size warnings; - unified catalog SQL executed against the migrated local PostgreSQL service and returned a valid empty page; - rebuilt local Docker backend and frontend images successfully; - backend, frontend, PostgreSQL, Neo4j, MinIO and Runner containers reported healthy; - `/api/system/health` returned application code `200`; - unauthenticated `/api/rules/catalog` returned HTTP `401`. Browser automation opened the rebuilt deployment and verified the local login surface. The documented local sample administrator password did not match the persisted administrator in this long-lived Docker volume, so authenticated Standard/Data Flow navigation was not forced by overwriting or resetting credentials. The build, component contracts, authenticated API tests, real PostgreSQL query, and container health checks provide the Task 8 acceptance evidence; an authenticated visual walkthrough can be repeated during the phase-level M4 acceptance with the environment owner’s current local credential. ## Review closeout The post-implementation review found eight convergence gaps. All eight are closed: 1. idempotency is emitted only for `rule.apply`; `standard.enforce` and `quality.check` now match the closed DataFlow contract; 2. catalog compatibility is never inferred from an arbitrary non-empty object; absent production-line context remains explicitly `unknown`; 3. `POST /api/interface/data/standard/code` is closed with HTTP 410 and read-only-migration semantics; 4. Standard add/update rejects executable code and server-attests any `rule_version_id` against the complete trusted publication evidence chain; 5. DataFlow drafts receive a server-owned UUIDv7 before assembly and saving fails closed if no identity can be obtained; 6. governed DataFlow create/update validates a closed envelope, verifies all referenced versions are published, preserves the immutable identity, and never invokes task-list, generated-script, n8n-workflow, or legacy data product side effects; 7. Standard authoring no longer fabricates a client-side `validated` state; new Standard clauses must bind a server-published RuleVersion; 8. generation evidence state `ready` is rendered as successful and async station values are reloaded by the existing deep watcher. The cutover deliberately preserves legacy non-governed records and read-only migration displays, while preventing those fields from becoming new governed execution semantics. ## Security review closeout The second review strengthened the boundary from server-issued identifiers to database-enforced, single-use draft reservations: - migration `20260723_210` adds a unique DataFlow UID, actor foreign key, unique nonce digest, expiration and consumption ledger; - the draft endpoint persists a short-lived reservation and returns a closed receipt; governed creation atomically consumes it using one conditional `UPDATE ... RETURNING`; - UID, actor, nonce digest, expiry and unused state must all match, so self-created UUIDs, replay, cross-actor use, expiration and concurrent duplicate consumption fail closed; - an existing governed DataFlow requires the complete governed envelope on every update, keeps the existing UID and cannot be downgraded through `script_type`, script path, task, code or workflow fields; - any partial governed signal is classified as governed and rejected before legacy task/script/n8n side effects can run. Data Standard natural-language authoring now deterministically produces a Rule candidate. The prompt states that rule explicitly, the authoring agent repairs a model-produced Standard candidate, and the API independently checks the post-validation candidate before recording or signing it. New legacy Standard records require an exactly attested published RuleVersion; only an existing legacy record may be updated without one for read-only migration. Catalog compatibility is now evaluated server-side against the production line's current input and output schema references. It returns `compatible`, `incompatible`, or `unknown` with a reason and evidence. The frontend supplies this context, blocks unknown/incompatible stations, hydrates an already-fixed version through an exact-version endpoint even when it is outside the first catalog page, and uses a monotonically increasing request sequence so stale searches cannot overwrite newer context. Real PostgreSQL acceptance confirmed: - Alembic upgraded the persisted Docker database from revision 200 to `20260723_210 (head)`; - the reservation table exists and its actor foreign key rejects an unknown actor; - two independent concurrent database sessions consuming one receipt produce exactly one acceptance and one rejection; - replay, cross-actor and expired receipt consumption are rejected; - acceptance fixtures are removed after the test. ## Recoverable create and trusted-schema closeout The third review replaced the one-shot PostgreSQL-consume/Neo4j-create boundary with a recoverable create Saga. Historical migration 210 remains unchanged; forward-only migration `20260724_220` adds: - `reserved`, `creating`, `completed`, and `failed` states; - an expiring create lease, attempt counter and error code; - the Neo4j node ID, bounded result JSON and canonical SHA-256 digest; - create-started, failed, completed and updated timestamps. `begin_dataflow_create` now uses one conditional `UPDATE ... RETURNING` to claim a lease. The claim is committed before the graph side effect. Governed DataFlow creation installs the idempotent `DataFlow.uid` unique constraint, uses `MERGE` only on the server-reserved UID, and verifies the persisted name, UID, script type and canonical governed requirement without overwriting an existing node. Completion conditionally owns the lease and persists the exact response. A retry after response loss returns that stored response; an expired lease reclaims the same UID, observes the existing graph node and finalizes it. Name/UID or immutable-property conflicts fail closed. Catalog compatibility no longer trusts matching reference strings alone: - the current input/output references are resolved through `SchemaResolver`; - published rules compare the current snapshot hashes with the hashes frozen into their published logical plan; - compile and test evidence IDs must be tied to that same logical plan and hash set; - a Standard expands every fixed `standard_rule_bindings` entry and requires the same trusted rule plan/evidence/hash chain for every clause; - resolver failure returns `unknown`; same-reference hash drift and any bound Standard rule drift return `incompatible`; - reader evidence contains only stable IDs, hashes, snapshot IDs and source revisions. The Data Flow editor stores the reservation expiry, renews it when less than 60 seconds remain, and changes to a new server UID only when the server returns the explicit `draft_reservation_refresh_required` signal. It retries that safe pre-side-effect case once. Network errors and in-progress leases do not cause a new UID; a completed response-loss retry is replayed by the server. Additional acceptance evidence: - local PostgreSQL upgraded from 210 to `20260724_220 (head)`; - two real concurrent PostgreSQL sessions produced one claim and one `dataflow_create_in_progress`; completion replay returned the identical result, expired drafts were rejected, and an expired create lease was reclaimed as attempt 2; - real Neo4j installed the `data_flow_uid` unique constraint, two same-UID merges produced one node, and immutable drift was rejected; - fault-injection tests cover Neo4j failure after a durable PG claim, PostgreSQL finalize failure after Neo4j success, completed-response replay, conflict closure and lease recovery; - real PostgreSQL and Neo4j integration: `2 passed`; - rebuilt backend and frontend images report migration 230, healthy database and Neo4j checks, application health code 200, and frontend HTTP 200. ## Durable snapshot and orphan-reconciliation closeout The fourth review closed the remaining transaction and recovery boundaries: - catalog endpoints now treat `SchemaResolver` as an explicit read-through persistence boundary: successful context-aware GETs commit newly created snapshots before returning their IDs, and every error path rolls back; - real PostgreSQL/API acceptance issued two separate catalog requests, observed one stable snapshot ID, and queried that ID from an independent database session; - forward-only migration `20260724_230` adds the canonical create request, server-owned create intent and SHA-256 request digest without modifying migration 220; - first claim atomically freezes request, intent and digest; every later claim, completed replay and finalize must match that digest; - changing the name or any other canonical payload/intent value while reusing a receipt fails with `dataflow_create_request_conflict` instead of silently returning an older result; - governed creation uses the supplied English name or a deterministic UID-derived fallback and never calls non-deterministic translation before computing the request digest. `DataFlowCreateReconciler` and `python -m app.commands.reconcile_dataflow_creates` recover orphaned graph nodes without accepting a nonce or inline business payload: - dry-run is the default and reports bounded safe IDs, state, digest and attempt information without changing state or attempt count; - apply mode claims expired `creating` or retryable `failed` rows with `FOR UPDATE SKIP LOCKED` and an expiring lease; - canonical request digest, duplicated intent and DataFlow UID are verified before any Neo4j write; - the existing UID-only graph `MERGE` either finds the orphan or creates the missing node, verifies immutable properties, and then finalizes PostgreSQL; - corrupt intent rows receive a terminal integrity error and do not block valid rows; other failures are isolated per reservation and remain retryable; - two real concurrent reconcilers produced exactly one claim; - real PostgreSQL/Neo4j acceptance created a graph node, deliberately omitted PostgreSQL finalize, discarded the receipt, expired the lease, and then completed the Saga from stored intent with exactly one graph node; - the reconciliation CLI dry-run completed against the local database with an empty, non-mutating report. Fourth-review acceptance: - focused repository/API/Saga/cutover contracts: `65 passed`; - real PostgreSQL/API/Neo4j integration: `4 passed`; - full repository suite: `673 passed, 33 skipped, 59 subtests passed`. ## Fenced reconciliation and strict-intent closeout The fifth review removed the remaining concurrency and poison-record failure modes from orphan reconciliation: - apply mode is bounded by the requested total but claims only one row at a time, commits that claim, processes it and only then claims the next row; a slow first item therefore cannot leave a batch of later items waiting on leases that expire before their processing begins; - after the trusted intent is semantically validated and immediately before the Neo4j side effect, the reconciler conditionally renews the lease using the reservation ID and current lease token. Lost ownership prevents the side effect and is reported as `lease_recovery_required`; - reconciliation parsing is isolated per claimed row. Scalar, malformed-shape or oversized request/intent JSON becomes terminal `create_intent_integrity_failed` without aborting the next valid item; - semantic intent validation failures become terminal `create_intent_validation_failed`. Both terminal codes are excluded from later list and claim queries, while infrastructure failures remain retryable; - failure persistence uses `UPDATE ... RETURNING`; a stale worker that no longer owns the lease cannot report a false successful failure transition; - governed tag relationships now use one atomic Neo4j `MERGE (DataFlow)-[:LABEL]->(DataLabel)` path shared by normal creation and reconciliation. Governed tag failures are strict: PostgreSQL is not finalized, the Saga remains retryable, and a later retry completes without duplicate relationships. Legacy tag handling retains its previous best-effort behavior. Fifth-review acceptance: - focused Saga/repository/cutover contracts: `40 passed`; - real PostgreSQL/API/Neo4j integration: `3 passed`; - full repository run completed with `680 passed, 33 skipped, 59 subtests passed` and one unrelated existing publication-receipt assertion mismatch: the tampered receipt was correctly rejected as non-canonical encoding while that randomized test expected the word `signature`; - selected Ruff `F`, `I`, and `B` checks passed; - the production frontend build completed with zero errors and the same existing 20 console warnings; - the backend image was rebuilt from the reviewed source; backend and frontend containers are healthy, Alembic reports `20260724_230 (head)`, application health reports database and Neo4j healthy with code 200, and frontend HTTP returns 200. ## Residual scope - Production-line cross-station compatibility remains ultimately authoritative at server-side release resolution; the UI presents current bound schema evidence and blocks on missing evidence rather than claiming runtime success. - Data Factory activation, canary, rollback, deployment permissions and environment operations remain Task 9 scope. - The repository has no published catalog fixtures in the current persisted local database, so catalog empty state was the live deployment state during this acceptance.