DataOps-platform/.superpowers/sdd/task-8-report.md

Task 8 Report — Standard / Data Flow product convergence

Date: 2026-07-24 Branch: codex/data-rule-execution-m3a-m5

Outcome

Task 8 removes the user-facing parallel rule implementations from Data Standard and Data Flow authoring.

The governed path is now:

1. describe a constraint in natural language against fixed input/output schema context;
2. review AI assumptions, ambiguities, model/prompt provenance and the signed generation receipt;
3. create an immutable draft RuleVersion;
4. run server-owned logical compilation and isolated sample testing;
5. publish only after trusted evidence succeeds;
6. select a published fixed RuleVersion or StandardVersion from one catalog;
7. assemble those fixed version IDs into ordered production-line stations;
8. hand the resulting DataFlowSpec to the release/Data Factory boundary.

Generated source code is never editable in the new path. Legacy code and free-text flow rules remain visible only in explicit migration sections and are excluded from new execution semantics.

Trusted catalog and evidence API

The control plane now exposes:

• GET /api/rules/catalog
  • closed query, asset_type, limit, and offset parameters;
  • rules and standards in one paginated contract;
  • human-readable name, stable asset UID, fixed version, owner, status, schema binding/scope, impact count, compiler backend, and latest compile/test evidence;
  • only published rules with an exact published logical plan, matching compile evidence, matching test evidence, and a publication audit;
  • only published standards whose fixed rule bindings are still published.
• GET /api/rules/catalog/assets/<rule|standard>/<version_id>/evidence
  • safe generation, logical compilation, dry-run, publication and physical plan stages;
  • IDs, hashes, lifecycle states, test kinds, and run IDs only;
  • no source text, generated candidate body, artifacts, samples, or unbounded evidence documents.

Compatibility aliases remain available at /api/rules/catalog/rule-versions and /api/rules/rule-versions/<version_id>/evidence. Global API policy keeps all GET paths behind rules:read.

AI authoring lifecycle

RuleAuthoringPanel now:

• blocks interpretation until input and output context are present;
• displays clarification requirements rather than guessing;
• shows assumptions, unresolved ambiguities, confidence, model/provider, prompt revision, repair count, and generation receipt status;
• creates a draft with the signed generation receipt;
• exposes explicit compile, isolated test and publish actions;
• enforces rules:edit and rules:publish UI gates;
• displays the trusted evidence timeline;
• never renders or edits generated code or a serialized candidate body.

Data Standard convergence

Data Standard editing now links a published RuleVersion through RuleCatalogPicker. New saves contain the fixed rule_version_id and bounded migration_metadata; they do not require or write an operation-code field.

Existing operation code is shown in a read-only migration area with clear unmigrated / linked state and a path back to natural-language re-authoring. It cannot become execution semantics merely by saving the legacy record.

Data Flow production-line assembly

Data Flow no longer edits free-text rules or embeds rule definitions. ProductionLineAssembler models the flow as ordered factory stations:

• standard.enforce fixes a standard_version_id;
• rule.apply and quality.check fix a rule_version_id;
• input/output selections are shown as dataset edges;
• schema-compatibility and latest-evidence summaries are shown per station;
• missing UID, dataset edges, fixed assets, compatibility, or evidence become explicit release-readiness blockers;
• reorder, remove and keyboard-accessible catalog selection are supported.

The new saved script_requirement envelope contains only:

• dataflow_spec;
• dataset_edges;
• bounded migration_metadata.

It contains no inline rule definition or generated source. Existing legacy requirements remain read-only and are not copied into that envelope.

UX and accessibility

The implementation reuses Vue 2, Vuetify, Material Design Icons, and the existing request client. No dependency or design system was added.

The interface uses a low-motion, medium/high-density governance-workbench style. Catalog loading, empty and error states are explicit. Catalog items support focus, arrow navigation, Enter/Space selection, listbox semantics, ARIA labels, and skeleton loading. Responsive layout collapses the Data Flow dataset grid on narrow screens.

Acceptance evidence

Fail-first:

• the initial frontend contract run produced 5 failed, 3 passed because the three governed components and convergence behavior did not exist.

Final:

• Task 8 API/repository/frontend and legacy-cutover contracts: 90 passed, 3 skipped, 59 subtests passed;
• full repository suite: 666 passed, 30 skipped, 59 subtests passed;
• changed Python Ruff: All checks passed!;
• git diff --check: passed;
• frontend production build: completed with 0 errors;
• build retained 20 pre-existing no-console warnings plus existing CSS ordering and bundle-size warnings;
• unified catalog SQL executed against the migrated local PostgreSQL service and returned a valid empty page;
• rebuilt local Docker backend and frontend images successfully;
• backend, frontend, PostgreSQL, Neo4j, MinIO and Runner containers reported healthy;
• /api/system/health returned application code 200;
• unauthenticated /api/rules/catalog returned HTTP 401.

Browser automation opened the rebuilt deployment and verified the local login surface. The documented local sample administrator password did not match the persisted administrator in this long-lived Docker volume, so authenticated Standard/Data Flow navigation was not forced by overwriting or resetting credentials. The build, component contracts, authenticated API tests, real PostgreSQL query, and container health checks provide the Task 8 acceptance evidence; an authenticated visual walkthrough can be repeated during the phase-level M4 acceptance with the environment owner’s current local credential.

Review closeout

The post-implementation review found eight convergence gaps. All eight are closed:

1. idempotency is emitted only for rule.apply; standard.enforce and quality.check now match the closed DataFlow contract;
2. catalog compatibility is never inferred from an arbitrary non-empty object; absent production-line context remains explicitly unknown;
3. POST /api/interface/data/standard/code is closed with HTTP 410 and read-only-migration semantics;
4. Standard add/update rejects executable code and server-attests any rule_version_id against the complete trusted publication evidence chain;
5. DataFlow drafts receive a server-owned UUIDv7 before assembly and saving fails closed if no identity can be obtained;
6. governed DataFlow create/update validates a closed envelope, verifies all referenced versions are published, preserves the immutable identity, and never invokes task-list, generated-script, n8n-workflow, or legacy data product side effects;
7. Standard authoring no longer fabricates a client-side validated state; new Standard clauses must bind a server-published RuleVersion;
8. generation evidence state ready is rendered as successful and async station values are reloaded by the existing deep watcher.

The cutover deliberately preserves legacy non-governed records and read-only migration displays, while preventing those fields from becoming new governed execution semantics.

Security review closeout

The second review strengthened the boundary from server-issued identifiers to database-enforced, single-use draft reservations:

• migration 20260723_210 adds a unique DataFlow UID, actor foreign key, unique nonce digest, expiration and consumption ledger;
• the draft endpoint persists a short-lived reservation and returns a closed receipt; governed creation atomically consumes it using one conditional UPDATE ... RETURNING;
• UID, actor, nonce digest, expiry and unused state must all match, so self-created UUIDs, replay, cross-actor use, expiration and concurrent duplicate consumption fail closed;
• an existing governed DataFlow requires the complete governed envelope on every update, keeps the existing UID and cannot be downgraded through script_type, script path, task, code or workflow fields;
• any partial governed signal is classified as governed and rejected before legacy task/script/n8n side effects can run.

Data Standard natural-language authoring now deterministically produces a Rule candidate. The prompt states that rule explicitly, the authoring agent repairs a model-produced Standard candidate, and the API independently checks the post-validation candidate before recording or signing it. New legacy Standard records require an exactly attested published RuleVersion; only an existing legacy record may be updated without one for read-only migration.

Catalog compatibility is now evaluated server-side against the production line's current input and output schema references. It returns compatible, incompatible, or unknown with a reason and evidence. The frontend supplies this context, blocks unknown/incompatible stations, hydrates an already-fixed version through an exact-version endpoint even when it is outside the first catalog page, and uses a monotonically increasing request sequence so stale searches cannot overwrite newer context.

Real PostgreSQL acceptance confirmed:

• Alembic upgraded the persisted Docker database from revision 200 to 20260723_210 (head);
• the reservation table exists and its actor foreign key rejects an unknown actor;
• two independent concurrent database sessions consuming one receipt produce exactly one acceptance and one rejection;
• replay, cross-actor and expired receipt consumption are rejected;
• acceptance fixtures are removed after the test.

Residual scope

• Production-line cross-station compatibility remains ultimately authoritative at server-side release resolution; the UI presents current bound schema evidence and blocks on missing evidence rather than claiming runtime success.
• Data Factory activation, canary, rollback, deployment permissions and environment operations remain Task 9 scope.
• The repository has no published catalog fixtures in the current persisted local database, so catalog empty state was the live deployment state during this acceptance.