DataOps-platform/.superpowers/sdd/task-8-report.md

Task 8 Report — Standard / Data Flow product convergence

Date: 2026-07-24 Branch: codex/data-rule-execution-m3a-m5

Outcome

Task 8 removes the user-facing parallel rule implementations from Data Standard and Data Flow authoring.

The governed path is now:

1. describe a constraint in natural language against fixed input/output schema context;
2. review AI assumptions, ambiguities, model/prompt provenance and the signed generation receipt;
3. create an immutable draft RuleVersion;
4. run server-owned logical compilation and isolated sample testing;
5. publish only after trusted evidence succeeds;
6. select a published fixed RuleVersion or StandardVersion from one catalog;
7. assemble those fixed version IDs into ordered production-line stations;
8. hand the resulting DataFlowSpec to the release/Data Factory boundary.

Generated source code is never editable in the new path. Legacy code and free-text flow rules remain visible only in explicit migration sections and are excluded from new execution semantics.

Trusted catalog and evidence API

The control plane now exposes:

• GET /api/rules/catalog
  • closed query, asset_type, limit, and offset parameters;
  • rules and standards in one paginated contract;
  • human-readable name, stable asset UID, fixed version, owner, status, schema binding/scope, impact count, compiler backend, and latest compile/test evidence;
  • only published rules with an exact published logical plan, matching compile evidence, matching test evidence, and a publication audit;
  • only published standards whose fixed rule bindings are still published.
• GET /api/rules/catalog/assets/<rule|standard>/<version_id>/evidence
  • safe generation, logical compilation, dry-run, publication and physical plan stages;
  • IDs, hashes, lifecycle states, test kinds, and run IDs only;
  • no source text, generated candidate body, artifacts, samples, or unbounded evidence documents.

Compatibility aliases remain available at /api/rules/catalog/rule-versions and /api/rules/rule-versions/<version_id>/evidence. Global API policy keeps all GET paths behind rules:read.

AI authoring lifecycle

RuleAuthoringPanel now:

• blocks interpretation until input and output context are present;
• displays clarification requirements rather than guessing;
• shows assumptions, unresolved ambiguities, confidence, model/provider, prompt revision, repair count, and generation receipt status;
• creates a draft with the signed generation receipt;
• exposes explicit compile, isolated test and publish actions;
• enforces rules:edit and rules:publish UI gates;
• displays the trusted evidence timeline;
• never renders or edits generated code or a serialized candidate body.

Data Standard convergence

Data Standard editing now links a published RuleVersion through RuleCatalogPicker. New saves contain the fixed rule_version_id and bounded migration_metadata; they do not require or write an operation-code field.

Existing operation code is shown in a read-only migration area with clear unmigrated / linked state and a path back to natural-language re-authoring. It cannot become execution semantics merely by saving the legacy record.

Data Flow production-line assembly

Data Flow no longer edits free-text rules or embeds rule definitions. ProductionLineAssembler models the flow as ordered factory stations:

• standard.enforce fixes a standard_version_id;
• rule.apply and quality.check fix a rule_version_id;
• input/output selections are shown as dataset edges;
• schema-compatibility and latest-evidence summaries are shown per station;
• missing UID, dataset edges, fixed assets, compatibility, or evidence become explicit release-readiness blockers;
• reorder, remove and keyboard-accessible catalog selection are supported.

The new saved script_requirement envelope contains only:

• dataflow_spec;
• dataset_edges;
• bounded migration_metadata.

It contains no inline rule definition or generated source. Existing legacy requirements remain read-only and are not copied into that envelope.

UX and accessibility

The implementation reuses Vue 2, Vuetify, Material Design Icons, and the existing request client. No dependency or design system was added.

The interface uses a low-motion, medium/high-density governance-workbench style. Catalog loading, empty and error states are explicit. Catalog items support focus, arrow navigation, Enter/Space selection, listbox semantics, ARIA labels, and skeleton loading. Responsive layout collapses the Data Flow dataset grid on narrow screens.

Acceptance evidence

Fail-first:

• the initial frontend contract run produced 5 failed, 3 passed because the three governed components and convergence behavior did not exist.

Final:

• Task 8 API/repository/frontend, Saga and legacy-cutover contracts: 64 passed;
• full repository suite: 672 passed, 31 skipped, 59 subtests passed;
• git diff --check: passed;
• frontend production build: completed with 0 errors;
• build retained 20 pre-existing no-console warnings plus existing CSS ordering and bundle-size warnings;
• unified catalog SQL executed against the migrated local PostgreSQL service and returned a valid empty page;
• rebuilt local Docker backend and frontend images successfully;
• backend, frontend, PostgreSQL, Neo4j, MinIO and Runner containers reported healthy;
• /api/system/health returned application code 200;
• unauthenticated /api/rules/catalog returned HTTP 401.

Browser automation opened the rebuilt deployment and verified the local login surface. The documented local sample administrator password did not match the persisted administrator in this long-lived Docker volume, so authenticated Standard/Data Flow navigation was not forced by overwriting or resetting credentials. The build, component contracts, authenticated API tests, real PostgreSQL query, and container health checks provide the Task 8 acceptance evidence; an authenticated visual walkthrough can be repeated during the phase-level M4 acceptance with the environment owner’s current local credential.

Review closeout

The post-implementation review found eight convergence gaps. All eight are closed:

1. idempotency is emitted only for rule.apply; standard.enforce and quality.check now match the closed DataFlow contract;
2. catalog compatibility is never inferred from an arbitrary non-empty object; absent production-line context remains explicitly unknown;
3. POST /api/interface/data/standard/code is closed with HTTP 410 and read-only-migration semantics;
4. Standard add/update rejects executable code and server-attests any rule_version_id against the complete trusted publication evidence chain;
5. DataFlow drafts receive a server-owned UUIDv7 before assembly and saving fails closed if no identity can be obtained;
6. governed DataFlow create/update validates a closed envelope, verifies all referenced versions are published, preserves the immutable identity, and never invokes task-list, generated-script, n8n-workflow, or legacy data product side effects;
7. Standard authoring no longer fabricates a client-side validated state; new Standard clauses must bind a server-published RuleVersion;
8. generation evidence state ready is rendered as successful and async station values are reloaded by the existing deep watcher.

The cutover deliberately preserves legacy non-governed records and read-only migration displays, while preventing those fields from becoming new governed execution semantics.

Security review closeout

The second review strengthened the boundary from server-issued identifiers to database-enforced, single-use draft reservations:

• migration 20260723_210 adds a unique DataFlow UID, actor foreign key, unique nonce digest, expiration and consumption ledger;
• the draft endpoint persists a short-lived reservation and returns a closed receipt; governed creation atomically consumes it using one conditional UPDATE ... RETURNING;
• UID, actor, nonce digest, expiry and unused state must all match, so self-created UUIDs, replay, cross-actor use, expiration and concurrent duplicate consumption fail closed;
• an existing governed DataFlow requires the complete governed envelope on every update, keeps the existing UID and cannot be downgraded through script_type, script path, task, code or workflow fields;
• any partial governed signal is classified as governed and rejected before legacy task/script/n8n side effects can run.

Data Standard natural-language authoring now deterministically produces a Rule candidate. The prompt states that rule explicitly, the authoring agent repairs a model-produced Standard candidate, and the API independently checks the post-validation candidate before recording or signing it. New legacy Standard records require an exactly attested published RuleVersion; only an existing legacy record may be updated without one for read-only migration.

Catalog compatibility is now evaluated server-side against the production line's current input and output schema references. It returns compatible, incompatible, or unknown with a reason and evidence. The frontend supplies this context, blocks unknown/incompatible stations, hydrates an already-fixed version through an exact-version endpoint even when it is outside the first catalog page, and uses a monotonically increasing request sequence so stale searches cannot overwrite newer context.

Real PostgreSQL acceptance confirmed:

• Alembic upgraded the persisted Docker database from revision 200 to 20260723_210 (head);
• the reservation table exists and its actor foreign key rejects an unknown actor;
• two independent concurrent database sessions consuming one receipt produce exactly one acceptance and one rejection;
• replay, cross-actor and expired receipt consumption are rejected;
• acceptance fixtures are removed after the test.

Recoverable create and trusted-schema closeout

The third review replaced the one-shot PostgreSQL-consume/Neo4j-create boundary with a recoverable create Saga. Historical migration 210 remains unchanged; forward-only migration 20260724_220 adds:

• reserved, creating, completed, and failed states;
• an expiring create lease, attempt counter and error code;
• the Neo4j node ID, bounded result JSON and canonical SHA-256 digest;
• create-started, failed, completed and updated timestamps.

begin_dataflow_create now uses one conditional UPDATE ... RETURNING to claim a lease. The claim is committed before the graph side effect. Governed DataFlow creation installs the idempotent DataFlow.uid unique constraint, uses MERGE only on the server-reserved UID, and verifies the persisted name, UID, script type and canonical governed requirement without overwriting an existing node. Completion conditionally owns the lease and persists the exact response. A retry after response loss returns that stored response; an expired lease reclaims the same UID, observes the existing graph node and finalizes it. Name/UID or immutable-property conflicts fail closed.

Catalog compatibility no longer trusts matching reference strings alone:

• the current input/output references are resolved through SchemaResolver;
• published rules compare the current snapshot hashes with the hashes frozen into their published logical plan;
• compile and test evidence IDs must be tied to that same logical plan and hash set;
• a Standard expands every fixed standard_rule_bindings entry and requires the same trusted rule plan/evidence/hash chain for every clause;
• resolver failure returns unknown; same-reference hash drift and any bound Standard rule drift return incompatible;
• reader evidence contains only stable IDs, hashes, snapshot IDs and source revisions.

The Data Flow editor stores the reservation expiry, renews it when less than 60 seconds remain, and changes to a new server UID only when the server returns the explicit draft_reservation_refresh_required signal. It retries that safe pre-side-effect case once. Network errors and in-progress leases do not cause a new UID; a completed response-loss retry is replayed by the server.

Additional acceptance evidence:

• local PostgreSQL upgraded from 210 to 20260724_220 (head);
• two real concurrent PostgreSQL sessions produced one claim and one dataflow_create_in_progress; completion replay returned the identical result, expired drafts were rejected, and an expired create lease was reclaimed as attempt 2;
• real Neo4j installed the data_flow_uid unique constraint, two same-UID merges produced one node, and immutable drift was rejected;
• fault-injection tests cover Neo4j failure after a durable PG claim, PostgreSQL finalize failure after Neo4j success, completed-response replay, conflict closure and lease recovery;
• real PostgreSQL and Neo4j integration: 2 passed;
• rebuilt backend and frontend images report migration 220, healthy database and Neo4j checks, application health code 200, and frontend HTTP 200.

Residual scope

• Production-line cross-station compatibility remains ultimately authoritative at server-side release resolution; the UI presents current bound schema evidence and blocks on missing evidence rather than claiming runtime success.
• Data Factory activation, canary, rollback, deployment permissions and environment operations remain Task 9 scope.
• The repository has no published catalog fixtures in the current persisted local database, so catalog empty state was the live deployment state during this acceptance.