docs: plan connector metadata boundary

docs/superpowers/plans/2026-08-19-enterprise-connector-metadata-boundary.md

@@ -0,0 +1,562 @@
+# Enterprise Connector and REST Catalog Boundary Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.
+
+**Goal:** Move database enterprise connectors into Data Factory, add a real PostgreSQL connector, and integrate REST Catalog into the Metadata page without changing existing connector APIs or security boundaries.
+
+**Architecture:** PostgreSQL becomes a versioned read-only catalog connector that reuses the current data-source manager and pool. The combined frontend is split into one allowlist-driven operations component, a database-only Data Factory wrapper, and a REST-only Metadata tab. The existing enterprise-connector route path and backend APIs remain stable.
+
+**Tech Stack:** Python 3, Flask, SQLAlchemy, pytest, Vue 2, Vuetify, Vuex, JavaScript, Docker Compose.
+
+---
+
+## File Structure
+
+- Create app/core/connectors/builtin/postgresql.py and its byte-identical deployment mirror for the PostgreSQL connector.
+- Modify both builtin/__init__.py files to register PostgreSQL.
+- Create tests/test_postgresql_enterprise_connector.py for backend behavior.
+- Modify frontend/src/router/routes.js for navigation ownership.
+- Create tests/test_enterprise_connector_metadata_navigation_contract.py for exact route ownership and order.
+- Create frontend/src/components/connectors/ConnectorOperations.vue for allowlist-driven shared operations.
+- Simplify frontend/src/views/dataGovernance/development/enterpriseConnectors.vue into the database wrapper.
+- Move the current metadata implementation into components/metadataManagement.vue.
+- Create components/externalCatalogAccess.vue and a new metadata/index.vue tab container.
+- Create tests/test_enterprise_connector_metadata_frontend_contract.py for filtering, tabs, and permission behavior.
+
+---
+
+### Task 1: Add a Real PostgreSQL Enterprise Connector
+
+**Files:**
+- Create: app/core/connectors/builtin/postgresql.py
+- Create: deployment/app/core/connectors/builtin/postgresql.py
+- Modify: app/core/connectors/builtin/__init__.py
+- Modify: deployment/app/core/connectors/builtin/__init__.py
+- Test: tests/test_postgresql_enterprise_connector.py
+
+- [ ] **Step 1: Write the failing backend test**
+
+Create a fake SQLAlchemy-compatible connection provider and verify registration, projection, scope, checkpoints, cancellation, resume, and secret-free evidence.
+
+~~~python
+from contextlib import contextmanager
+
+import pytest
+
+from app.core.connectors.builtin import register_builtin_connectors
+from app.core.connectors.builtin.postgresql import PostgreSQLConnector
+from app.core.connectors.errors import ConnectorCancelledError
+from app.core.connectors.registry import ConnectorRegistry
+from app.core.connectors.sdk import OperationRequest
+
+
+ROWS = [
+    {
+        "schema_name": "public",
+        "asset_name": "orders",
+        "asset_type": "BASE TABLE",
+        "column_name": "id",
+        "ordinal_position": 1,
+        "data_type": "uuid",
+        "is_nullable": "NO",
+        "column_default": None,
+        "column_comment": "order identifier",
+    },
+    {
+        "schema_name": "internal",
+        "asset_name": "jobs",
+        "asset_type": "BASE TABLE",
+        "column_name": "id",
+        "ordinal_position": 1,
+        "data_type": "bigint",
+        "is_nullable": "NO",
+        "column_default": None,
+        "column_comment": None,
+    },
+]
+
+
+class Result:
+    def mappings(self):
+        return self
+
+    def all(self):
+        return ROWS
+
+
+class Connection:
+    def execute(self, statement, parameters):
+        assert "information_schema.columns" in str(statement)
+        assert parameters == {}
+        return Result()
+
+
+@contextmanager
+def connection_provider(source_uid, purpose, **kwargs):
+    assert source_uid == "source-postgres"
+    assert purpose == "metadata_collection"
+    assert kwargs == {}
+    yield Connection()
+
+
+def request(operation="discover", **changes):
+    values = {
+        "source_uid": "source-postgres",
+        "operation": operation,
+        "config": {"credential_ref": "env:DATAOPS_CONNECTOR_POSTGRES"},
+        "scope": {"include_schemas": ["public"]},
+        "checkpoint": {},
+    }
+    values.update(changes)
+    return OperationRequest(**values)
+
+
+def test_postgresql_connector_registers_and_projects_catalog():
+    registry = register_builtin_connectors(
+        ConnectorRegistry(), connection_provider=connection_provider
+    )
+    connector = registry.resolve("postgresql", "1.0.0")
+    assert isinstance(connector, PostgreSQLConnector)
+    assert connector.manifest.display_name == "PostgreSQL"
+    assert connector.manifest.capabilities == (
+        "discover", "snapshot", "incremental", "cancel", "resume", "evidence"
+    )
+    result = connector.discover(request())
+    assert [item["asset_key"] for item in result.records] == [
+        "source-postgres:public.orders"
+    ]
+    assert result.checkpoint["snapshot_summary"]["record_count"] == 1
+    assert result.evidence["query_kind"] == "read_only_metadata"
+    assert "DATAOPS_CONNECTOR_POSTGRES" not in str(result)
+
+
+def test_postgresql_connector_incremental_cancel_resume_and_evidence():
+    connector = PostgreSQLConnector(connection_provider)
+    first = connector.snapshot(request("snapshot"))
+    incremental = connector.incremental(
+        request("incremental", checkpoint=first.checkpoint)
+    )
+    assert incremental.evidence["diff"]["changed"] is False
+    assert connector.resume(request("resume", checkpoint=first.checkpoint)).status == "succeeded"
+    assert connector.cancel(request("cancel")).status == "cancelled"
+    assert connector.evidence(request("evidence")).evidence["secret_material"] is False
+    with pytest.raises(ConnectorCancelledError):
+        connector.discover(request(cancel_probe=lambda: True))
+~~~
+
+- [ ] **Step 2: Run RED**
+
+Run:
+
+~~~bash
+PYTHONPATH=. .venv/bin/pytest -q tests/test_postgresql_enterprise_connector.py
+~~~
+
+Expected: collection fails because app.core.connectors.builtin.postgresql does not exist.
+
+- [ ] **Step 3: Implement PostgreSQL**
+
+Create PostgreSQLConnector as a ReadOnlyCatalogConnector. Use information_schema.columns joined to information_schema.tables and pg_catalog description tables; exclude pg_catalog and information_schema. Use connector_id postgresql, version 1.0.0, display name PostgreSQL, DATABASE_CONFIG_SCHEMA, and capabilities discover/snapshot/incremental/cancel/resume/evidence. health() and compatibility() check importlib.util.find_spec("psycopg2").
+
+Register it between Oracle and SQL Server:
+
+~~~python
+connection_provider = dependencies.get("connection_provider")
+registry.register(OracleConnector(connection_provider))
+registry.register(PostgreSQLConnector(connection_provider))
+registry.register(SqlServerConnector(connection_provider))
+~~~
+
+Add PostgreSQLConnector to __all__. Copy both changed backend files byte-for-byte into deployment/app.
+
+- [ ] **Step 4: Run GREEN and mirrors**
+
+~~~bash
+PYTHONPATH=. .venv/bin/pytest -q   tests/test_postgresql_enterprise_connector.py   tests/test_phase3_wp03_enterprise_connectors.py
+cmp app/core/connectors/builtin/postgresql.py deployment/app/core/connectors/builtin/postgresql.py
+cmp app/core/connectors/builtin/__init__.py deployment/app/core/connectors/builtin/__init__.py
+~~~
+
+Expected: all tests pass and both comparisons exit 0.
+
+- [ ] **Step 5: Commit**
+
+~~~bash
+git add app/core/connectors/builtin/postgresql.py   deployment/app/core/connectors/builtin/postgresql.py   app/core/connectors/builtin/__init__.py   deployment/app/core/connectors/builtin/__init__.py   tests/test_postgresql_enterprise_connector.py
+git commit -m "feat: add postgresql enterprise connector"
+~~~
+
+---
+
+### Task 2: Move Enterprise Connectors into Data Factory
+
+**Files:**
+- Modify: frontend/src/router/routes.js
+- Create: tests/test_enterprise_connector_metadata_navigation_contract.py
+
+- [ ] **Step 1: Write the failing navigation test**
+
+Use the scoped route-block pattern from tests/test_enterprise_identity_navigation_contract.py. Assert:
+
+~~~python
+def test_enterprise_connectors_belong_only_to_data_factory():
+    routes = ROUTES.read_text(encoding="utf-8")
+    research = route_block(routes, "data-governance", 4)
+    factory = route_block(routes, "dataFactory", 4)
+    assert routes.count("name: 'enterpriseConnectors'") == 1
+    assert "name: 'enterpriseConnectors'" not in research
+    assert "name: 'enterpriseConnectors'" in factory
+    block = route_block(factory, "enterpriseConnectors", 8)
+    assert "path: '/data-governance/development/enterprise-connectors'" in block
+    assert "component: 'dataGovernance/development/enterpriseConnectors'" in block
+    assert "permissions: ['connectors:read']" in block
+
+
+def test_data_factory_order_keeps_legacy_entries_independent():
+    routes = ROUTES.read_text(encoding="utf-8")
+    factory = route_block(routes, "dataFactory", 4)
+    names = [
+        "enterpriseConnectors",
+        "productionLineDeployment",
+        "productionLine",
+        "workflowIndex",
+        "dataObservability",
+        "connectionPoolIndex",
+    ]
+    positions = [factory.index(f"name: '{name}'") for name in names]
+    assert positions == sorted(positions)
+~~~
+
+- [ ] **Step 2: Run RED**
+
+~~~bash
+PYTHONPATH=. .venv/bin/pytest -q tests/test_enterprise_connector_metadata_navigation_contract.py
+~~~
+
+Expected: enterpriseConnectors is still under Data Research.
+
+- [ ] **Step 3: Move the route**
+
+Move the existing object unchanged into dataFactory.children immediately before productionLineDeployment. Add sort: 4. Preserve path, name, component, icon, label, and connectors:read. Do not change n8n paths, targets, names, or components.
+
+- [ ] **Step 4: Run GREEN**
+
+~~~bash
+PYTHONPATH=. .venv/bin/pytest -q   tests/test_enterprise_connector_metadata_navigation_contract.py   tests/test_data_factory_orchestration_navigation_contract.py   tests/test_enterprise_identity_menu_filter_contract.py
+~~~
+
+Expected: all pass.
+
+- [ ] **Step 5: Commit**
+
+~~~bash
+git add frontend/src/router/routes.js   tests/test_enterprise_connector_metadata_navigation_contract.py
+git commit -m "feat: move enterprise connectors to data factory"
+~~~
+
+---
+
+### Task 3: Extract the Database Connector Operations Surface
+
+**Files:**
+- Create: frontend/src/components/connectors/ConnectorOperations.vue
+- Modify: frontend/src/views/dataGovernance/development/enterpriseConnectors.vue
+- Create: tests/test_enterprise_connector_metadata_frontend_contract.py
+
+- [ ] **Step 1: Write failing contracts**
+
+~~~python
+def test_shared_operations_is_allowlist_driven_and_fail_closed():
+    source = SHARED.read_text(encoding="utf-8")
+    assert "connectorIds" in source
+    assert "normalizedConnectorIds" in source
+    assert "filter(item => allowed.has(item.connector_id))" in source
+    assert "if (!this.normalizedConnectorIds.length)" in source
+
+
+def test_enterprise_surface_is_database_only():
+    source = DATABASE.read_text(encoding="utf-8")
+    assert "['oracle', 'postgresql', 'sqlserver']" in source
+    assert "当前开放范围仅限数据库访问" in source
+    assert "文件目录、对象存储、API 与消息系统" in source
+    assert "rest-catalog" not in source
+~~~
+
+- [ ] **Step 2: Run RED**
+
+~~~bash
+PYTHONPATH=. .venv/bin/pytest -q tests/test_enterprise_connector_metadata_frontend_contract.py
+~~~
+
+Expected: the shared component is missing and the old page still mixes REST Catalog.
+
+- [ ] **Step 3: Implement ConnectorOperations.vue**
+
+Move the current manifest cards, run table, compatibility, Dry-run, cancel, and resume behavior into the shared component. Add strict props:
+
+~~~javascript
+props: {
+  connectorIds: {
+    type: Array,
+    required: true,
+    validator: value => value.length > 0 && value.every(
+      item => typeof item === 'string' && /^[a-z][a-z0-9_-]{2,63}$/.test(item)
+    )
+  },
+  mode: {
+    type: String,
+    required: true,
+    validator: value => ['database', 'rest-catalog'].includes(value)
+  }
+}
+~~~
+
+Normalize the allowlist and fail closed:
+
+~~~javascript
+normalizedConnectorIds () {
+  if (!Array.isArray(this.connectorIds)) return []
+  return [...new Set(this.connectorIds.filter(
+    item => typeof item === 'string' && /^[a-z][a-z0-9_-]{2,63}$/.test(item)
+  ))]
+}
+~~~
+
+In loadAll(), return without API calls when the list is empty. Otherwise load manifests and runs, then filter both with the same Set. Clear stale arrays in the catch block. Render base_url and allowed_host only in rest-catalog mode. Preserve existing connectors:operate/manage checks and human Dry-run rules.
+
+- [ ] **Step 4: Make enterpriseConnectors.vue a database wrapper**
+
+Render the heading, the enterprise-UAT warning, and:
+
+~~~vue
+<v-alert type="info" outlined>
+  当前开放范围仅限数据库访问。文件目录、对象存储、API 与消息系统等来源将在后续版本扩展开发。
+</v-alert>
+<connector-operations
+  :connector-ids="['oracle', 'postgresql', 'sqlserver']"
+  mode="database"
+/>
+~~~
+
+- [ ] **Step 5: Run GREEN and build**
+
+~~~bash
+PYTHONPATH=. .venv/bin/pytest -q   tests/test_enterprise_connector_metadata_frontend_contract.py   tests/test_phase3_wp03_enterprise_connectors.py
+npm --prefix frontend run build
+~~~
+
+Expected: tests pass and build exits 0.
+
+- [ ] **Step 6: Commit**
+
+~~~bash
+git add frontend/src/components/connectors/ConnectorOperations.vue   frontend/src/views/dataGovernance/development/enterpriseConnectors.vue   tests/test_enterprise_connector_metadata_frontend_contract.py
+git commit -m "refactor: isolate database connector operations"
+~~~
+
+---
+
+### Task 4: Integrate REST Catalog into Metadata Governance
+
+**Files:**
+- Move: frontend/src/views/dataGovernance/metadata/index.vue
+- Create: frontend/src/views/dataGovernance/metadata/components/metadataManagement.vue
+- Create: frontend/src/views/dataGovernance/metadata/components/externalCatalogAccess.vue
+- Create: frontend/src/views/dataGovernance/metadata/index.vue
+- Modify: tests/test_enterprise_connector_metadata_frontend_contract.py
+
+- [ ] **Step 1: Add failing tab and permission tests**
+
+~~~python
+def test_metadata_has_management_and_external_catalog_tabs():
+    source = METADATA.read_text(encoding="utf-8")
+    assert source.count("<v-tab") == 2
+    assert "元数据管理" in source
+    assert "外部目录接入" in source
+    assert "permissions.includes('connectors:read')" in source
+    assert 'v-if="canReadConnectors"' in source
+
+
+def test_external_catalog_is_rest_only_and_management_is_preserved():
+    external = EXTERNAL.read_text(encoding="utf-8")
+    management = MANAGEMENT.read_text(encoding="utf-8")
+    assert "['rest-catalog']" in external
+    assert 'mode="rest-catalog"' in external
+    assert "Oracle" not in external and "SQL Server" not in external
+    for marker in ("新增元数据", "handleSubmit", "handleDelete", "getMetaDataList"):
+        assert marker in management
+~~~
+
+- [ ] **Step 2: Run RED**
+
+~~~bash
+PYTHONPATH=. .venv/bin/pytest -q tests/test_enterprise_connector_metadata_frontend_contract.py
+~~~
+
+Expected: the tab wrapper and REST component are missing.
+
+- [ ] **Step 3: Preserve metadata management**
+
+Move the complete current index.vue into components/metadataManagement.vue. Only fix relative imports caused by the move:
+
+~~~javascript
+import FilterList from '../../components/Filter'
+import Edit from './edit'
+import AddAudit from './addAudit'
+~~~
+
+Keep its existing CRUD, audit, status, dialogs, assistant comment, and Data Research Center link unchanged.
+
+- [ ] **Step 4: Create externalCatalogAccess.vue**
+
+~~~vue
+<template>
+  <div class="pt-4">
+    <v-alert type="info" outlined>
+      REST Catalog 用于从受控 HTTPS 元数据目录同步资产，不采集任意业务 API 数据。
+    </v-alert>
+    <v-alert type="warning" outlined>
+      真实企业目录地址、允许域名、凭据和 UAT 仍需外部提供。
+    </v-alert>
+    <connector-operations :connector-ids="['rest-catalog']" mode="rest-catalog" />
+  </div>
+</template>
+~~~
+
+Import ConnectorOperations and register it as the only child component.
+
+- [ ] **Step 5: Recreate metadata/index.vue as a permission-aware wrapper**
+
+~~~vue
+<template>
+  <div class="pa-3 white">
+    <v-tabs v-model="tab" show-arrows>
+      <v-tab>元数据管理</v-tab>
+      <v-tab v-if="canReadConnectors">外部目录接入</v-tab>
+    </v-tabs>
+    <v-tabs-items v-model="tab">
+      <v-tab-item><metadata-management /></v-tab-item>
+      <v-tab-item v-if="canReadConnectors"><external-catalog-access /></v-tab-item>
+    </v-tabs-items>
+  </div>
+</template>
+~~~
+
+Compute permissions from this.$store.state.user.userInfo.permissions and return permissions.includes('connectors:read') from canReadConnectors. The v-if must prevent mounting and API loading for unauthorized users.
+
+- [ ] **Step 6: Run GREEN and build**
+
+~~~bash
+PYTHONPATH=. .venv/bin/pytest -q   tests/test_enterprise_connector_metadata_frontend_contract.py   tests/data_research/test_data_research_frontend_contract.py   tests/test_phase3_wp03_enterprise_connectors.py   tests/test_frontend_rbac_contract.py
+npm --prefix frontend run build
+~~~
+
+Expected: all pass and build exits 0.
+
+- [ ] **Step 7: Commit**
+
+~~~bash
+git add frontend/src/views/dataGovernance/metadata/index.vue   frontend/src/views/dataGovernance/metadata/components/metadataManagement.vue   frontend/src/views/dataGovernance/metadata/components/externalCatalogAccess.vue   tests/test_enterprise_connector_metadata_frontend_contract.py
+git commit -m "feat: integrate rest catalog with metadata governance"
+~~~
+
+---
+
+### Task 5: Run Affected Regression and Static Gates
+
+**Files:** Verification only unless a failure is proven to be caused by Tasks 1–4.
+
+- [ ] **Step 1: Run the complete affected suite**
+
+~~~bash
+PYTHONPATH=. .venv/bin/pytest -q   tests/test_postgresql_enterprise_connector.py   tests/test_enterprise_connector_metadata_navigation_contract.py   tests/test_enterprise_connector_metadata_frontend_contract.py   tests/test_phase3_wp03_enterprise_connectors.py   tests/data_research/test_data_research_frontend_contract.py   tests/test_data_factory_orchestration_navigation_contract.py   tests/test_enterprise_identity_menu_filter_contract.py   tests/test_frontend_rbac_contract.py   tests/test_architecture_artifacts.py
+~~~
+
+Expected: all collected tests pass and this change introduces no skips.
+
+- [ ] **Step 2: Run existing real PostgreSQL integration**
+
+~~~bash
+PYTHONPATH=. .venv/bin/pytest -q tests/integration/test_phase3_wp03_connectors_postgres.py
+~~~
+
+Expected: pass using the repository's documented Docker test database. If the fixture reports the database environment is unavailable, record the exact blocker and do not report this command as passed.
+
+- [ ] **Step 3: Run build, compile, mirror, and diff gates**
+
+~~~bash
+npm --prefix frontend run build
+.venv/bin/python -m py_compile   app/core/connectors/builtin/postgresql.py   deployment/app/core/connectors/builtin/postgresql.py   tests/test_postgresql_enterprise_connector.py   tests/test_enterprise_connector_metadata_navigation_contract.py   tests/test_enterprise_connector_metadata_frontend_contract.py
+cmp app/core/connectors/builtin/postgresql.py deployment/app/core/connectors/builtin/postgresql.py
+cmp app/core/connectors/builtin/__init__.py deployment/app/core/connectors/builtin/__init__.py
+git diff --check
+git status --short
+~~~
+
+Expected: all commands exit 0. Preserve existing untracked architecture assets and do not stage them.
+
+- [ ] **Step 4: Commit only proven gate corrections**
+
+If an affected contract required a real update, stage its exact files and commit:
+
+~~~bash
+git commit -m "test: align connector metadata boundary contracts"
+~~~
+
+Do not create an empty commit.
+
+---
+
+### Task 6: Deploy and Perform Local UAT
+
+**Files:** No source changes expected.
+
+- [ ] **Step 1: Rebuild local backend and frontend**
+
+~~~bash
+docker compose -f deploy/docker/docker-compose.yml up -d --build backend frontend
+~~~
+
+Expected: build exits 0 and frontend/backend become healthy.
+
+- [ ] **Step 2: Verify services and deep links**
+
+~~~bash
+docker compose -f deploy/docker/docker-compose.yml ps
+curl -fsS http://localhost:15500/api/system/health
+curl -fsS http://localhost:18183/data-governance/development/enterprise-connectors >/dev/null
+curl -fsS http://localhost:18183/data-governance/metadata >/dev/null
+~~~
+
+Expected: health reports status=healthy and both SPA links return 200.
+
+- [ ] **Step 3: Perform authenticated browser UAT**
+
+Verify all nine conditions:
+
+1. Data Research no longer lists 企业连接器.
+2. Data Factory lists 企业连接器 before 数据生产线投产.
+3. Enterprise Connectors shows exactly Oracle, PostgreSQL, and SQL Server.
+4. The database-only scope notice is visible.
+5. REST Catalog is absent from that page.
+6. Metadata shows 元数据管理 and 外部目录接入 for connectors:read.
+7. External Catalog Access shows REST Catalog and no database connectors.
+8. Both direct links survive refresh.
+9. A metadata user without connectors:read retains metadata management and does not mount external-catalog content.
+
+Do not execute a connector against a real external system during this navigation UAT. Compatibility or Dry-run requires a separately approved test endpoint and credential.
+
+- [ ] **Step 4: Run final independent review**
+
+PASS requires zero Critical and zero Important findings. Fix each valid finding with a new RED/GREEN cycle and repeat affected review.
+
+- [ ] **Step 5: Record final branch state**
+
+~~~bash
+git log --oneline -8
+git status --short
+git diff --check
+~~~
+
+Expected: feature files committed, unrelated untracked user assets untouched, and no push or production deployment without separate instruction.
+