DataOps-platform/docs/superpowers/plans/2026-07-29-wp06-device-entity-resolution.md

WP-06 Device Entity Resolution and Reversible Merge Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Generate explainable cross-source device-match candidates, govern manual or tightly gated automatic merges, and preserve complete rollback evidence without deleting source assets or source identities.

Architecture: PostgreSQL remains authoritative for match candidates, immutable reviews, merge events, and rollback events. A merge is a non-destructive canonical/member relationship: WP-04 asset rows, versions, and source mappings remain unchanged, while the active merge event determines the canonical asset and its grouped source identities. Deterministic rules generate the first production-ready candidates; governed AI proposals may enter the same lifecycle with provider, model, evidence, and confidence, but cannot auto-merge in WP-06.

Tech Stack: Flask, SQLAlchemy, PostgreSQL JSONB, Alembic, Vue 2, Vuetify, pytest.

---

Global Constraints

• Candidate pairs must have the same asset type, come from different source systems, and contain two distinct active assets.
• Pair identity is normalized by sorted asset UID so the same pair cannot have two simultaneously open candidates.
• Rule scoring uses independently explainable signals: normalized name, location, organization, responsible person, source code, and configured model attribute.
• Candidate generation is bounded to at most 500 assets and 2,000 new candidates per request.
• Rule candidates require platform asset and source-mapping evidence. AI candidates additionally require provider, model, confidence in [0, 1], and evidence UIDs.
• Credentials, tokens, connection strings, personal contact fields, and raw operating measurements are rejected and never returned.
• Viewer may read. Editor may generate deterministic candidates and submit governed AI candidates. Only admin with device-entities:review plus the unique accountable asset_manager for device_mapping/DEVICE_ENTITY_RESOLUTION may approve, reject, auto-merge, or roll back.
• Automatic merge is disabled by default. When explicitly enabled it is rule-only, requires confidence at or above 0.98, requires the initiating actor to pass the accountable-manager gate, and writes the same review and merge evidence as a manual approval.
• A canonical asset may have multiple members. A member may belong to only one active canonical group, and a canonical asset may not itself be an active member.
• Merge never rewrites or deletes device_assets, device_asset_source_mappings, or device_asset_versions.
• Rollback appends an immutable rollback event; it does not delete or rewrite the merge event.
• WP-06 does not perform survivorship updates to asset attributes, physical deduplication, quality scoring, root-cause analysis, or downstream graph projection.
• Validation follows the user-approved rule: run only WP-06 domain, persistence, API, permission, frontend, migration, contract, and local browser checks; do not run the full repository regression.
• Continue on codex/dataops-phase1-equipment-governance; do not push or deploy remotely.

Task 1: Deterministic Candidates and Governed Decisions

Files:

• Create: app/core/data_research/device_entity_resolution.py
• Modify: app/core/data_research/errors.py
• Create: tests/data_research/test_device_entity_resolution.py

Interfaces:

• Produces records DeviceEntityCandidateRecord, DeviceEntityReviewRecord, DeviceEntityMergeRecord, and DeviceEntityRollbackRecord.
• Produces score_device_pair(left, right), DeviceEntityResolutionService.generate, submit_ai_candidate, search, get, review, rollback, reviews, merges, and rollbacks.

• Repository boundary supplies assets with source mappings, open-pair lookup, candidate persistence, row locks, active canonical/member lookup, and immutable evidence append methods.

• [x] Step 1: Write failing scoring and lifecycle tests

Cover hand-derived score literals, Unicode/case/spacing normalization, same-source and different-type rejection, bounded generation, duplicate-open-candidate suppression, required evidence, AI-provider requirements, secret rejection, manual approval, accountable-manager rejection, automatic-merge default-off, strict rule-only automatic merge, canonical/member conflicts, immutable reviews, and append-only rollback.

• Step 2: Verify RED

Run:

PYTHONPATH=. .venv/bin/pytest -q \
  tests/data_research/test_device_entity_resolution.py

Expected: collection fails because device_entity_resolution does not exist.

• Step 3: Implement the minimal domain service

Implement deterministic normalization and signal scoring without external model calls. Store explanations as a list of {signal, matched, weight, left, right} entries and evidence as bounded UID lists. Treat approval as an atomic review plus non-destructive merge event; reject invalid transitions and stale versions with 409.

• Step 4: Verify GREEN

Run the Task 1 test command and expect all domain tests to pass.

Task 2: PostgreSQL Candidate, Merge, and Rollback Ledger

Files:

• Create: app/core/data_research/device_entity_repository.py
• Modify: app/models/data_research.py
• Create: migrations/versions/20260729_310_device_entity_resolution.py
• Modify: tests/test_database_migrations.py
• Create: tests/integration/test_device_entity_resolution_postgres.py

Interfaces:

• Produces SqlAlchemyDeviceEntityResolutionRepository.
• Adds device_entity_match_candidates, device_entity_match_reviews, device_entity_merge_events, and device_entity_merge_rollbacks.
• Candidate status is pending|merged|rejected|rolled_back; suggestion source is rule|ai|manual; review decision is approve|reject|auto_approve.

• Active membership is a merge event without a corresponding rollback event.

• [x] Step 1: Write failing migration and real-PostgreSQL tests

Cover database constraints, pair lookup, bounded filters, row locks, multiple members under one canonical asset, single active canonical per member, merge snapshots, immutable review records, immutable rollback records, and cleanup limited to test-owned rows.

• Step 2: Verify RED

Run:

TEST_DATABASE_URL=postgresql+psycopg2://dataops:dataops-test-password@127.0.0.1:15432/dataops \
PYTHONPATH=. .venv/bin/pytest -q \
  tests/test_database_migrations.py::test_device_entity_resolution_migration_is_non_destructive_and_reversible \
  tests/integration/test_device_entity_resolution_postgres.py

Expected: failure because the migration and repository do not exist.

• Step 3: Implement migration, models, and repository

Use UUIDv7 identities, JSONB evidence/explanations/snapshots, a partial unique index for one open pair, transactional active-member conflict checks, and FOR UPDATE for review and rollback. Keep downgrade data-preserving.

• Step 4: Upgrade local PostgreSQL and verify GREEN

Upgrade the isolated local database to 20260729_310, then rerun the Task 2 command with TEST_DATABASE_URL.

Task 3: Permission-Controlled Entity-Resolution API

Files:

• Modify: app/api/data_development/routes.py
• Modify: app/core/system/permissions.py
• Create: tests/data_research/test_device_entity_resolution_api.py
• Modify: tests/test_permission_matrix.py

Interfaces:

• Produces:

  • GET /api/development/v1/device-entities/candidates
  • POST /api/development/v1/device-entities/candidates/generate
  • POST /api/development/v1/device-entities/candidates
  • GET /api/development/v1/device-entities/candidates/{candidate_uid}
  • POST /api/development/v1/device-entities/candidates/{candidate_uid}/review
  • GET /api/development/v1/device-entities/candidates/{candidate_uid}/reviews
  • GET /api/development/v1/device-entities/candidates/{candidate_uid}/merges
  • POST /api/development/v1/device-entities/merges/{merge_uid}/rollback
  • GET /api/development/v1/device-entities/merges/{merge_uid}/rollbacks

• [x] Step 1: Write failing API and permission tests

Cover viewer reads, viewer write denial, editor generation/AI submission, editor review denial, admin platform permission plus accountable-manager runtime denial, safe bounded filters, automatic-merge feature flag serialization, review/merge/rollback evidence, and safe error envelopes.

• Step 2: Verify RED

Run:

PYTHONPATH=. .venv/bin/pytest -q \
  tests/data_research/test_device_entity_resolution_api.py \
  tests/test_permission_matrix.py

Expected: failure because the routes and dedicated permissions do not exist.

• Step 3: Implement API and authorization

Add device-entities:edit to editor/admin and device-entities:review to admin. Reads remain on governance:read; /review and /rollback require the review permission, while the service repeats the unique accountable-manager runtime check for device_mapping/DEVICE_ENTITY_RESOLUTION.

• Step 4: Verify GREEN

Run the Task 3 command and expect all tests to pass.

Task 4: Entity-Resolution Workbench and Delivery Evidence

Files:

• Modify: frontend/src/api/dataDevelopment.js
• Modify: frontend/src/router/routes.js
• Modify: frontend/src/views/dataGovernance/development/index.vue
• Create: frontend/src/views/dataGovernance/development/deviceEntityResolution.vue
• Create: frontend/src/views/dataGovernance/development/deviceEntityResolutionModel.js
• Create: frontend/tests/device-entity-resolution-model.test.mjs
• Modify: docs/architecture/OPENAPI.yaml
• Modify: docs/architecture/DATA_MODEL.md
• Modify: docs/FUNCTION_MODULE_CENSUS_20260726.md
• Modify: docs/DATAOPS_PHASE1_3_MONTH_WORK_PLAN_20260729.md
• Modify: deployment/app/ for the WP-06 backend subset.

Interfaces:

• Produces /data-governance/development/entity-resolution with candidate status filters, explainable signal comparison, evidence display, generate action, permission-aware approve/reject controls, canonical-asset selection, merge history, and rollback evidence.

• [x] Step 1: Write failing frontend model tests

Cover status and source labels, confidence formatting, matched-signal summary, canonical/member selection, and permission-aware review/rollback decisions.

• Step 2: Verify RED

Run:

cd frontend
node --test tests/device-entity-resolution-model.test.mjs

Expected: failure because the frontend model module does not exist.

• Step 3: Implement the workbench

Add the development-center entry and responsive workbench. Keep candidate evidence and lifecycle visible to viewers; hide mutation actions without permissions; display that AI candidates always require manual review and that physical deduplication/survivorship remains outside WP-06.

• Step 4: Regenerate contracts and update ledgers

Run:

.venv/bin/python scripts/generate_openapi.py \
  --output docs/architecture/OPENAPI.yaml

Record SEM-15 and SEM-17 as engineering complete pending enterprise data acceptance. Record SEM-16 as partially built because deterministic high-confidence auto-merge is feature-gated and governed AI auto-merge remains intentionally disabled.

• Step 5: Run targeted verification

Run only the WP-06 domain, PostgreSQL, API, permission, migration, OpenAPI, frontend model, targeted lint, production build, release-copy parity, and local browser workflow. In the browser generate a cross-source candidate, inspect its score explanation, approve it as the accountable asset manager, confirm grouped identities, roll it back, and confirm zero console errors. Run git diff --check.

• Step 6: Commit

Create one independently reversible WP-06 engineering commit. Do not push.