# Task 8 Report — Standard / Data Flow product convergence

Date: 2026-07-24
Branch: `codex/data-rule-execution-m3a-m5`

## Outcome

Task 8 removes the user-facing parallel rule implementations from Data
Standard and Data Flow authoring.

The governed path is now:

1. describe a constraint in natural language against fixed input/output
   schema context;
2. review AI assumptions, ambiguities, model/prompt provenance and the signed
   generation receipt;
3. create an immutable draft RuleVersion;
4. run server-owned logical compilation and isolated sample testing;
5. publish only after trusted evidence succeeds;
6. select a published fixed RuleVersion or StandardVersion from one catalog;
7. assemble those fixed version IDs into ordered production-line stations;
8. hand the resulting DataFlowSpec to the release/Data Factory boundary.

Generated source code is never editable in the new path. Legacy code and
free-text flow rules remain visible only in explicit migration sections and
are excluded from new execution semantics.

## Trusted catalog and evidence API

The control plane now exposes:

- `GET /api/rules/catalog`
  - closed `query`, `asset_type`, `limit`, and `offset` parameters;
  - rules and standards in one paginated contract;
  - human-readable name, stable asset UID, fixed version, owner, status,
    schema binding/scope, impact count, compiler backend, and latest
    compile/test evidence;
  - only published rules with an exact published logical plan, matching
    compile evidence, matching test evidence, and a publication audit;
  - only published standards whose fixed rule bindings are still published.
- `GET /api/rules/catalog/assets/<rule|standard>/<version_id>/evidence`
  - safe generation, logical compilation, dry-run, publication and physical
    plan stages;
  - IDs, hashes, lifecycle states, test kinds, and run IDs only;
  - no source text, generated candidate body, artifacts, samples, or
    unbounded evidence documents.

Compatibility aliases remain available at
`/api/rules/catalog/rule-versions` and
`/api/rules/rule-versions/<version_id>/evidence`. Global API policy keeps all
GET paths behind `rules:read`.

## AI authoring lifecycle

`RuleAuthoringPanel` now:

- blocks interpretation until input and output context are present;
- displays clarification requirements rather than guessing;
- shows assumptions, unresolved ambiguities, confidence, model/provider,
  prompt revision, repair count, and generation receipt status;
- creates a draft with the signed generation receipt;
- exposes explicit compile, isolated test and publish actions;
- enforces `rules:edit` and `rules:publish` UI gates;
- displays the trusted evidence timeline;
- never renders or edits generated code or a serialized candidate body.

## Data Standard convergence

Data Standard editing now links a published RuleVersion through
`RuleCatalogPicker`. New saves contain the fixed `rule_version_id` and bounded
`migration_metadata`; they do not require or write an operation-code field.

Existing operation code is shown in a read-only migration area with clear
`unmigrated` / `linked` state and a path back to natural-language
re-authoring. It cannot become execution semantics merely by saving the
legacy record.

## Data Flow production-line assembly

Data Flow no longer edits free-text rules or embeds rule definitions.
`ProductionLineAssembler` models the flow as ordered factory stations:

- `standard.enforce` fixes a `standard_version_id`;
- `rule.apply` and `quality.check` fix a `rule_version_id`;
- input/output selections are shown as dataset edges;
- schema-compatibility and latest-evidence summaries are shown per station;
- missing UID, dataset edges, fixed assets, compatibility, or evidence become
  explicit release-readiness blockers;
- reorder, remove and keyboard-accessible catalog selection are supported.

The new saved `script_requirement` envelope contains only:

- `dataflow_spec`;
- `dataset_edges`;
- bounded `migration_metadata`.

It contains no inline rule definition or generated source. Existing legacy
requirements remain read-only and are not copied into that envelope.

## UX and accessibility

The implementation reuses Vue 2, Vuetify, Material Design Icons, and the
existing request client. No dependency or design system was added.

The interface uses a low-motion, medium/high-density governance-workbench
style. Catalog loading, empty and error states are explicit. Catalog items
support focus, arrow navigation, Enter/Space selection, listbox semantics,
ARIA labels, and skeleton loading. Responsive layout collapses the Data Flow
dataset grid on narrow screens.

## Acceptance evidence

Fail-first:

- the initial frontend contract run produced `5 failed, 3 passed` because the
  three governed components and convergence behavior did not exist.

Final:

- Task 8 API/repository/frontend and legacy-cutover contracts:
  `90 passed, 3 skipped, 59 subtests passed`;
- full repository suite:
  `666 passed, 30 skipped, 59 subtests passed`;
- changed Python Ruff: `All checks passed!`;
- `git diff --check`: passed;
- frontend production build: completed with `0 errors`;
- build retained 20 pre-existing `no-console` warnings plus existing CSS
  ordering and bundle-size warnings;
- unified catalog SQL executed against the migrated local PostgreSQL service
  and returned a valid empty page;
- rebuilt local Docker backend and frontend images successfully;
- backend, frontend, PostgreSQL, Neo4j, MinIO and Runner containers reported
  healthy;
- `/api/system/health` returned application code `200`;
- unauthenticated `/api/rules/catalog` returned HTTP `401`.

Browser automation opened the rebuilt deployment and verified the local login
surface. The documented local sample administrator password did not match the
persisted administrator in this long-lived Docker volume, so authenticated
Standard/Data Flow navigation was not forced by overwriting or resetting
credentials. The build, component contracts, authenticated API tests, real
PostgreSQL query, and container health checks provide the Task 8 acceptance
evidence; an authenticated visual walkthrough can be repeated during the
phase-level M4 acceptance with the environment owner’s current local
credential.

## Review closeout

The post-implementation review found eight convergence gaps. All eight are
closed:

1. idempotency is emitted only for `rule.apply`; `standard.enforce` and
   `quality.check` now match the closed DataFlow contract;
2. catalog compatibility is never inferred from an arbitrary non-empty
   object; absent production-line context remains explicitly `unknown`;
3. `POST /api/interface/data/standard/code` is closed with HTTP 410 and
   read-only-migration semantics;
4. Standard add/update rejects executable code and server-attests any
   `rule_version_id` against the complete trusted publication evidence chain;
5. DataFlow drafts receive a server-owned UUIDv7 before assembly and saving
   fails closed if no identity can be obtained;
6. governed DataFlow create/update validates a closed envelope, verifies all
   referenced versions are published, preserves the immutable identity, and
   never invokes task-list, generated-script, n8n-workflow, or legacy data
   product side effects;
7. Standard authoring no longer fabricates a client-side `validated` state;
   new Standard clauses must bind a server-published RuleVersion;
8. generation evidence state `ready` is rendered as successful and async
   station values are reloaded by the existing deep watcher.

The cutover deliberately preserves legacy non-governed records and
read-only migration displays, while preventing those fields from becoming
new governed execution semantics.

## Security review closeout

The second review strengthened the boundary from server-issued identifiers
to database-enforced, single-use draft reservations:

- migration `20260723_210` adds a unique DataFlow UID, actor foreign key,
  unique nonce digest, expiration and consumption ledger;
- the draft endpoint persists a short-lived reservation and returns a closed
  receipt; governed creation atomically consumes it using one conditional
  `UPDATE ... RETURNING`;
- UID, actor, nonce digest, expiry and unused state must all match, so
  self-created UUIDs, replay, cross-actor use, expiration and concurrent
  duplicate consumption fail closed;
- an existing governed DataFlow requires the complete governed envelope on
  every update, keeps the existing UID and cannot be downgraded through
  `script_type`, script path, task, code or workflow fields;
- any partial governed signal is classified as governed and rejected before
  legacy task/script/n8n side effects can run.

Data Standard natural-language authoring now deterministically produces a
Rule candidate. The prompt states that rule explicitly, the authoring agent
repairs a model-produced Standard candidate, and the API independently checks
the post-validation candidate before recording or signing it. New legacy
Standard records require an exactly attested published RuleVersion; only an
existing legacy record may be updated without one for read-only migration.

Catalog compatibility is now evaluated server-side against the production
line's current input and output schema references. It returns
`compatible`, `incompatible`, or `unknown` with a reason and evidence.
The frontend supplies this context, blocks unknown/incompatible stations,
hydrates an already-fixed version through an exact-version endpoint even
when it is outside the first catalog page, and uses a monotonically
increasing request sequence so stale searches cannot overwrite newer
context.

Real PostgreSQL acceptance confirmed:

- Alembic upgraded the persisted Docker database from revision 200 to
  `20260723_210 (head)`;
- the reservation table exists and its actor foreign key rejects an unknown
  actor;
- two independent concurrent database sessions consuming one receipt produce
  exactly one acceptance and one rejection;
- replay, cross-actor and expired receipt consumption are rejected;
- acceptance fixtures are removed after the test.

## Residual scope

- Production-line cross-station compatibility remains ultimately authoritative
  at server-side release resolution; the UI presents current bound schema
  evidence and blocks on missing evidence rather than claiming runtime success.
- Data Factory activation, canary, rollback, deployment permissions and
  environment operations remain Task 9 scope.
- The repository has no published catalog fixtures in the current persisted
  local database, so catalog empty state was the live deployment state during
  this acceptance.