DataOps-platform/docs/phase3/P3_WP12_METERING_SHOWBACK_IMPLEMENTATION_PLAN.md

P3-WP12 Metering and Showback Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Build a reconcilable, explainable, local-engineering-only metering and Showback baseline without enabling financial settlement or chargeback.

Architecture: app/core/governance/metering_showback.py owns closed normalization, Decimal/micros arithmetic, immutable correction chains, allocation replay, budget decisions, local outbox records and read models. PostgreSQL stores only normalized identifiers, safe evidence digest/reference and recomputable digests; metering_showback_runtime_* SECURITY DEFINER gateways form the sole runtime fact access path. The system blueprint derives principal/scope from persisted grants, emits no-store closed API responses, and never accepts tenant/domain/principal from a request body.

Tech Stack: Python 3.11, Flask, SQLAlchemy, PostgreSQL 15+, Alembic, Decimal, pytest, OpenAPI generator.

---

Boundary, prerequisites, and activation state

• The enterprise input cost remains exactly TBD_EXTERNAL; its owner is enterprise_finance_owner. The migration, fixture manifest, API and evidence must state ENGINEERING_EVIDENCE_ONLY and must never call a real billing, finance, mail or collaboration system.
• The only enabled provider is local-metering-fixture-v1@1. It emits deterministic, non-sensitive fixture events and a fixed safe local-fixture://wp12/v1 reference; no URL, raw row, credential, SQL, script or object payload is accepted or persisted.
• Chargeback is a DB, code and API fail-closed state: configuration is immutable enabled=false; all names and operations for receivable, invoice, deduction, payment and settlement are rejected before any write. A Showback report is informational only.
• The final engineering status may be ENGINEERING_BASELINE_COMPLETE_SHOWBACK_ACTIVATION_BLOCKED only after all local gates pass. Enterprise finance approval, cost-center master data, enterprise metering source, price book, budget and Showback decision remain activation blockers; this work does not assert UAT or production acceptance.

Read-only baseline (2026-08-18)

• Migration head is 20260818_545; WP12 appends 20260818_546 through 20260818_552, including integrity, event-time rollup, unified replay/legacy-ACL, allocation-atomicity and fixed-definer-owner follow-ups; it must not edit prior revisions.

Execution record (2026-08-18)

• First REDs were recorded for the missing core contract, migration, Flask/OpenAPI route and, in the follow-up, missing allocation/budget controls and deployment migration mirror.
• GREEN delivered 20260818_546 facts/runtime gateway, 20260818_547 allocation/budget SECURITY DEFINER control gateway, and 20260818_548 persistent rule-version replay, with source/deployment byte parity.
• Controlled Docker PostgreSQL evidence covers isolated 545→548, real Flask commits through distinct control/runtime roles, concurrent exact-once disabled outbox, persisted rule replay, lease/fence/restart/ACL/cross-domain/Chargeback negative paths, and shared empty 547→548→547→548 role-init replay. Full historical empty-database replay remains separately blocked at the pre-existing WP06 restricted-migrator gate.
• Independent review follow-up added 549: event-time persisted Showback rollup, correction/reference scope guard, duplicate/zero-target and overlap rejection, plus exact OpenAPI query/nested-body contracts. Controlled PostgreSQL passed isolated 545→549 and shared empty 548→549→548→549 role-init replay.
• Follow-up review added 550: replay is now event-time coverage-consistent with Showback/reconciliation, strict for a specified partial/expired rule, old renamed entrypoints have no runtime execute privilege, and a nonempty legacy-data preflight refuses activation. Controlled PostgreSQL covered isolated 545→550, legacy 548→549→550 rejection and shared empty 549→550→549→550 replay.
• Final quality follow-up added 551: all three real Flask POST paths preserve the three-field server-derived mapping boundary, return generic no-store 4xx for gateway/database failures, and make event lease+write atomic so a rejected changed replay cannot strand a lease. Allocation publication now serializes same-rule changed/exact replays, has a deferred total-weight database guard and a btree_gist exclusion constraint for concurrent overlapping scope windows. Controlled PostgreSQL covered two-connection Barrier races and empty shared 550→551→550→551 replay.
• Owner-fence follow-up added 552: every WP12 SECURITY DEFINER function, including the total-weight trigger and renamed legacy gateways, is transferred to the fixed dataops_tenant_foundation_owner NOLOGIN role and has PUBLIC, app/runtime and control execution revoked before current gateway grants are restored. The migration fail-closes unless role-init has supplied the non-login/non-superuser/non-CREATEROLE owner and its required schema capability. Controlled PostgreSQL covered a temporary NOSUPER/NOCREATEROLE migrator that owned the 551 functions, 551→552→551→552, owner census, role deletion and shared role-init replay.
• Reuse the WP11 database contract: restricted owner/control/runtime identities, fixed search_path, closed JSONB functions, explicit PUBLIC/dataops_app/runtime revokes, persisted scope grants, lease fence and request idempotency. Reuse WP10's server-derived scope and closed Flask body rules; do not let a client claim a tenant, business domain or principal.
• Source-of-truth files are app/ and migrations/; every touched source must have byte-identical counterparts under deployment/app/ and deployment/migrations/. OpenAPI is generated with scripts/generate_openapi.py and mirrored through its established deployment script.

File map

• Create: app/core/governance/metering_showback.py — contracts, normalizers, fixture manifest, digest/reconciliation and application service.
• Create: app/api/system/metering_showback_routes.py — closed Showback-only Flask endpoints.
• Modify: app/api/system/__init__.py — import the endpoint module.
• Modify: app/core/system/permissions.py — metering:read and metering:manage constants, role membership and deny-by-default path classifier.
• Create: migrations/versions/20260818_546_metering_showback.py — roles, tables, constraints, SECURITY DEFINER runtime/control/read functions and privilege fence; downgrade() refuses nonempty facts.
• Modify: docs/architecture/OPENAPI.yaml through scripts/generate_openapi.py — closed schemas, no-store, 201/200/400/403/409 and exact RBAC annotations.
• Create: docs/phase3/p3-wp12-metering-fixture/manifest.json — versioned deterministic engineering fixture and TBD_EXTERNAL enterprise declaration.
• Create: docs/runbooks/P3_WP12_METERING_SHOWBACK_OPERATIONS.md — safe operation, reconciliation and activation gates.
• Create: docs/validation/P3_WP12_FAILURE_INJECTION.json and docs/validation/P3_WP12_METERING_SHOWBACK_EVIDENCE.md — failure cases and command/result evidence.
• Modify: docs/phase3/P3_WP00_EXECUTION_REGISTERS.json, docs/phase3/P3_WP00_REQUIREMENTS.json, docs/DATAOPS_PHASE3_6_MONTH_DEVELOPMENT_PLAN_20260802.md — preserve the external cost prerequisite and record the local baseline, not enterprise activation.
• Create: tests/core/governance/test_wp12_metering_showback.py, tests/test_wp12_metering_showback_api.py, tests/test_wp12_metering_showback_migration_contract.py, tests/integration/test_wp12_metering_showback_postgres.py.
• Create exact deployment mirrors for every new/modified app/ and migration source and mirror the fixture/runbook/evidence only when the repository's deployment manifest requires it.

Task 1: Closed event contract and deterministic fixture

Files:

• Create: tests/core/governance/test_wp12_metering_showback.py
• Create: app/core/governance/metering_showback.py

• Create: docs/phase3/p3-wp12-metering-fixture/manifest.json

• [ ] Step 1: Write RED tests for the eight event kinds, closed object and safe evidence rules.

def test_normalize_event_accepts_only_eight_kinds_and_safe_digest_reference():
    event = normalize_event(FIXTURE_EVENT)
    assert event["event_kind"] == "query"
    assert event["evidence"] == {"digest": "a" * 64, "reference": "local-fixture://wp12/v1"}
    for invalid in ({"event_kind": "invoice"}, {"raw_row": "x"}, {"evidence": {"url": "https://x"}}):
        with pytest.raises(ValueError):
            normalize_event(FIXTURE_EVENT | invalid)

def test_decimal_conversion_rejects_float_nan_inf_bool_and_dimension_mismatch():
    assert to_micros("1.250000", "gb", "mb") == 1250_000
    for value in (1.25, True, "NaN", "Infinity"):
        with pytest.raises(ValueError):
            to_micros(value, "gb", "mb")
    with pytest.raises(ValueError):
        to_micros("1", "seconds", "gb")

• Step 2: Run RED.

Run: .venv/bin/pytest -q tests/core/governance/test_wp12_metering_showback.py -k 'normalize_event or decimal_conversion'

Expected: import/attribute failure for the absent WP12 module or functions, never an unrelated collection error.

• Step 3: Implement the minimal closed normalizer.

Implement immutable exact-key JSON object checks; permit only query, api, file, subscription, storage, compute, task, model_call; normalize Unicode with NFKC and reject changed/confusable identifiers; use Decimal(str) only for strings and store integer micros with the (unit, dimension) allowlist. Canonical JSON SHA-256 is the sole digest representation. Keep raw_payload_retained=False.

• Step 4: Run GREEN and save fixture manifest.

Run: .venv/bin/pytest -q tests/core/governance/test_wp12_metering_showback.py -k 'normalize_event or decimal_conversion'

Expected: all selected tests pass; manifest.json declares schema_version: 1, mode: ENGINEERING_EVIDENCE_ONLY, enterprise_cost_input: TBD_EXTERNAL, owner: enterprise_finance_owner, and only local fixture records.

Task 2: Idempotent facts, corrections, allocation replay and budget alert contract

Files:

• Modify: tests/core/governance/test_wp12_metering_showback.py

• Modify: app/core/governance/metering_showback.py

• [ ] Step 1: Write RED tests for exact replay/conflict, append-only late corrections, allocation and alert semantics.

def test_exact_replay_is_stable_but_changed_idempotency_payload_conflicts(service):
    first = service.record(FIXTURE_EVENT)
    assert service.record(FIXTURE_EVENT) == first
    with pytest.raises(ReplayConflict):
        service.record(FIXTURE_EVENT | {"quantity": "2.000000"})

def test_late_event_creates_append_only_correction_and_reconciles_source_to_allocated(service):
    original = service.record(FIXTURE_EVENT)
    correction = service.correct(original["event_uid"], LATE_FIXTURE_EVENT)
    assert correction["correction_of"] == original["event_uid"]
    assert service.reconcile("2026-08") == {"source_micros": 1250000, "allocated_micros": 1250000, "difference_micros": 0}

def test_allocation_rule_replay_uses_effective_version_and_exact_weights(service):
    rule = service.publish_allocation(RULE_100_PERCENT)
    assert service.replay_allocation(rule["rule_version"], "2026-08") == service.replay_allocation(rule["rule_version"], "2026-08")
    with pytest.raises(ValueError): service.publish_allocation(RULE_100_PERCENT | {"weight": "0.999999"})

def test_budget_alert_outbox_is_exact_once_and_chargeback_is_denied(service):
    assert service.evaluate_budget(BUDGET) ["alert_created"] is True
    assert service.evaluate_budget(BUDGET) ["alert_created"] is False
    with pytest.raises(PermissionError): service.request_chargeback({"enabled": True})

• Step 2: Run RED.

Run: .venv/bin/pytest -q tests/core/governance/test_wp12_metering_showback.py -k 'replay or correction or allocation or budget or chargeback'

Expected: missing application-service behavior fails before database code is added.

• Step 3: Implement pure application service behavior.

Implement an explicit repository protocol and in-memory adapter used only by unit tests. Key facts by (tenant_ref, event_uid) and protect an idempotency key with canonical request digest; changed request digest raises ReplayConflict. A late correction must create a new event referencing the prior UID, never mutate/delete the source event. Allocation versions carry half-open effective windows and Decimal weights whose exact sum is 1.000000; report source and allocated total micros plus difference. Budget thresholds create one durable-intent record keyed by budget/version/window/threshold; provider stays disabled and records contain no destination. request_chargeback always raises PermissionError("chargeback_disabled").

• Step 4: Run GREEN.

Run: .venv/bin/pytest -q tests/core/governance/test_wp12_metering_showback.py

Expected: all contract tests pass without network, filesystem fixture mutation or float arithmetic.

Task 3: PostgreSQL facts, gateways, roles and downgrade fence

Files:

• Create: tests/test_wp12_metering_showback_migration_contract.py
• Create: tests/integration/test_wp12_metering_showback_postgres.py

• Create: migrations/versions/20260818_546_metering_showback.py

• [ ] Step 1: Write RED migration-contract and PostgreSQL tests.

def test_wp12_migration_is_child_of_545_and_has_closed_runtime_gateway():
    source = MIGRATION.read_text()
    assert 'revision = "20260818_546"' in source and 'down_revision = "20260818_545"' in source
    assert "SECURITY DEFINER SET search_path=pg_catalog,public" in source
    assert "REVOKE ALL ON TABLE public.metering_events FROM PUBLIC,dataops_app,dataops_app_runtime" in source
    assert "chargeback_disabled" in source

@pytest.mark.integration
def test_postgres_runtime_is_gateway_only_and_fence_rejects_stale_writer(runtime_url):
    assert runtime_has_no_table_dml(runtime_url, "metering_events")
    first, stale = race_claim_then_expire_and_reclaim(runtime_url)
    assert stale["lease_fence"] < first["lease_fence"]
    with pytest.raises(Exception, match="metering_stale_fence"): runtime_write(runtime_url, stale)

• Step 2: Run RED.

Run: .venv/bin/pytest -q tests/test_wp12_metering_showback_migration_contract.py

Expected: the new migration path is absent and the source-contract test fails.

• Step 3: Add revision 20260818_546.

Create metering_scope_grants, metering_events, metering_allocation_rules, metering_allocations, metering_budgets, metering_alert_outbox, metering_reconciliation_reports, metering_audit_events, and metering_runtime_claims. Use primary/unique/check constraints for tenant/domain/project/cost-center mapping, event kinds, 64-hex digests, safe references, integer micros, event correction lineage, rule version/window, exact weight_micros total 1_000_000, immutable chargeback_enabled=false, and alert idempotency. Create required roles only when role-init pre-provisioned them; create owner/control/runtime functions with fixed search path, JSONB exact-key checks, advisory locking, clock-based lease expiry, monotonically increasing fence, persisted principal scope check and request digest. Revoke direct facts/sequence/function rights from PUBLIC, dataops_app and runtime; grant only metering_showback_runtime_write/read to runtime and claim issuance to control. downgrade() raises whenever any WP12 fact or alert exists; only empty schema can be removed.

• Step 4: Run source GREEN then real PostgreSQL when controlled DSNs are supplied.

Run: .venv/bin/pytest -q tests/test_wp12_metering_showback_migration_contract.py

Expected: migration contract passes.

Run: .venv/bin/pytest -q tests/integration/test_wp12_metering_showback_postgres.py

Expected: controlled PostgreSQL proves idempotency under two connections, restart/new-app visibility, role-init replay privilege revocation, stale lease fence denial, cross-scope rejection, direct-facts ACL denial, reconciliation equality and nonempty downgrade refusal. If TEST_DATABASE_URL/runtime/control DSNs are absent, the test must report SKIPPED and evidence must mark the real-PostgreSQL gate unexecuted rather than passed.

Task 4: Repository adapter and Flask/RBAC/OpenAPI contract

Files:

• Modify: app/core/governance/metering_showback.py
• Create: app/api/system/metering_showback_routes.py
• Modify: app/api/system/__init__.py
• Modify: app/core/system/permissions.py
• Create: tests/test_wp12_metering_showback_api.py
• Modify: scripts/generate_openapi.py

• Modify: docs/architecture/OPENAPI.yaml

• [ ] Step 1: Write RED HTTP/OpenAPI/RBAC tests.

def test_record_and_showback_are_closed_no_store_and_server_scoped(client, login):
    response = client.post("/api/system/metering/events", json={**API_EVENT, "tenant_ref": "spoof"})
    assert response.status_code == 400 and response.headers["Cache-Control"] == "no-store"
    allowed = client.post("/api/system/metering/events", json=API_EVENT, headers=login("metering:manage"))
    assert allowed.status_code == 201 and allowed.headers["Cache-Control"] == "no-store"

def test_read_cross_scope_and_chargeback_endpoint_are_denied(client, login):
    assert client.get("/api/system/metering/showback?window=2026-08", headers=login("other-scope")).status_code == 403
    assert client.post("/api/system/metering/chargeback", json={}, headers=login("metering:manage")).status_code in {403, 404}

def test_openapi_declares_closed_body_rbac_and_no_store():
    spec = yaml.safe_load(OPENAPI.read_text())
    assert spec["paths"]["/api/system/metering/events"]["post"]["x-required-permission"] == "metering:manage"
    assert spec["components"]["schemas"]["MeteringEventRequest"]["additionalProperties"] is False

• Step 2: Run RED.

Run: .venv/bin/pytest -q tests/test_wp12_metering_showback_api.py

Expected: routes/spec/permission constants do not yet exist, yielding expected missing-route or missing-schema failures.

• Step 3: Implement the closed API.

Register only POST /api/system/metering/events (201 and idempotent 200), POST /api/system/metering/allocations, POST /api/system/metering/budgets, GET /api/system/metering/showback, GET /api/system/metering/reconciliation, and GET /api/system/metering/audit. Every body is exact-key and bounded, all responses set Cache-Control: no-store, and every scope/principal comes from server authentication plus persisted grant. Do not add a chargeback route. Add METERING_READ and METERING_MANAGE only to appropriate roles, update the deny-by-default path map, and use a separate control connection for claim issuance with no fallback to the shared runtime login. Generate OpenAPI from decorators; preserve closed request schemas and documented 200/201/400/403/409 results.

• Step 4: Run GREEN and regenerate the spec.

Run: .venv/bin/pytest -q tests/test_wp12_metering_showback_api.py && .venv/bin/python scripts/generate_openapi.py && .venv/bin/pytest -q tests/test_wp12_metering_showback_api.py

Expected: API contract passes before and after generated OpenAPI; no cacheable response or client-supplied authorization field is accepted.

Task 5: Documentation, failure injection, source/deployment parity and final verification

Files:

• Create: docs/runbooks/P3_WP12_METERING_SHOWBACK_OPERATIONS.md
• Create: docs/validation/P3_WP12_FAILURE_INJECTION.json
• Create: docs/validation/P3_WP12_METERING_SHOWBACK_EVIDENCE.md
• Modify: docs/phase3/P3_WP00_EXECUTION_REGISTERS.json
• Modify: docs/phase3/P3_WP00_REQUIREMENTS.json
• Modify: docs/DATAOPS_PHASE3_6_MONTH_DEVELOPMENT_PLAN_20260802.md

• Create/modify: matching deployment/app/**, deployment/migrations/**, deployment/scripts/generate_openapi.py, and deployment manifest entries when applicable.

• [ ] Step 1: Write RED parity and documentation assertions.

def test_wp12_source_deployment_mirrors_and_activation_boundary():
    assert source_bytes("app/core/governance/metering_showback.py") == deployment_bytes("deployment/app/core/governance/metering_showback.py")
    assert source_bytes(MIGRATION) == deployment_bytes(DEPLOYMENT_MIGRATION)
    evidence = EVIDENCE.read_text()
    assert "ENGINEERING_EVIDENCE_ONLY" in evidence and "TBD_EXTERNAL" in evidence
    assert "enterprise_finance_owner" in evidence and "UAT passed" not in evidence

• Step 2: Run RED.

Run: .venv/bin/pytest -q tests/test_wp12_metering_showback_migration_contract.py -k 'mirror or activation'

Expected: absent deployment copies/evidence fail only because WP12 artifacts are not present.

• Step 3: Write operational artifacts and exact mirrors.

The runbook must document fixture-only ingestion, reconciliation query, append-only correction procedure, alert inspection without delivery, disabled chargeback behavior, lease/fence recovery and explicit finance activation prerequisites. Failure JSON must cover bad Unicode/credentials/raw URL, changed replay, late correction, unit mismatch, allocation non-1, duplicate alert, cross-tenant/domain, direct DML, stale fence, role-init replay, nonempty downgrade and chargeback. Evidence records each RED and GREEN command/result, only claims real PostgreSQL where the test actually ran, and labels all fixture records local. Update registers and master plan without changing cost from TBD_EXTERNAL.

• Step 4: Synchronize then run final targeted verification.

Run: diff -u app/core/governance/metering_showback.py deployment/app/core/governance/metering_showback.py && diff -u app/api/system/metering_showback_routes.py deployment/app/api/system/metering_showback_routes.py && diff -u migrations/versions/20260818_546_metering_showback.py deployment/migrations/versions/20260818_546_metering_showback.py

Expected: no diff.

Run: .venv/bin/pytest -q tests/core/governance/test_wp12_metering_showback.py tests/test_wp12_metering_showback_api.py tests/test_wp12_metering_showback_migration_contract.py

Expected: all targeted non-integration tests pass.

Run: .venv/bin/pytest -q tests/integration/test_wp12_metering_showback_postgres.py

Expected: PostgreSQL tests pass only with controlled DSNs; otherwise all relevant cases skip and evidence retains the gate as unexecuted.

Review checklist

• Eight and only eight metering event kinds; bounded exact Decimal/micros and compatible unit dimensions.
• Idempotency exact replay, changed-payload conflict, and append-only late correction are covered by RED/GREEN evidence.
• Allocation mapping carries department, business domain, project and cost center; rule versions/effective windows replay deterministically and reconcile exactly.
• Budget/threshold creates durable local alert intent once; no provider, mail, collaboration message, enterprise billing or financial artifact exists.
• Runtime cannot directly read or mutate facts; control/runtime identities, request claims, lease/fence and role-init replay are tested against PostgreSQL.
• API/RBAC/no-store/OpenAPI is closed and cross-scope denied; no chargeback endpoint or bypass is exposed.
• Downgrade refuses nonempty facts; app/deployment source bytes match.
• All enterprise cost inputs remain TBD_EXTERNAL; completion language is limited to local engineering evidence.