DataOps-platform/.superpowers/sdd/task-5-report.md

Task 5 Report — Polars cross-source compilation and artifact I/O

Date: 2026-07-23 Branch: codex/data-rule-execution-m3a-m5 Lifecycle boundary: physical Polars plans are created as compiled; this task does not add a production promotion API. The real execution test inserts a published fixture only to exercise the pre-existing Runner publication gate.

Outcome

Implemented a closed JSON Polars compiler, shared compile/runtime semantic validation, allowlisted LazyFrame reconstruction, server-owned digest-bound Parquet artifact storage, correlation-scoped PostgreSQL artifact handoff, current output-binding re-attestation, separated operation metrics, Runner registration, canonical plan attestations, and a real PostgreSQL + MySQL + MinIO integration through the production repository/resolver/executor path.

The plan contains canonical RuleVersion, SchemaSnapshot, and DatasetBinding hashes plus dataops-polars-1.42.1 provenance. It contains no Python source, pickle, callable, module, client path, arbitrary URL, secret, or Polars internal serialized plan.

TDD evidence

RED evidence was captured before each production slice:

• Compiler module: 5 failed because app.core.data_rules.compilers.polars did not exist.
• ArtifactStore and adapter modules: 8 failed because the modules did not exist.
• Cross-source compiler registry/service: 2 failed because the registry rejected cross-source artifacts.
• Runner canonical attestation/correlation: 1 failed because correlation was not forwarded.
• Node registry trusted context: 1 failed because correlation_id was not accepted.
• Artifact describe/bootstrap configuration: 2 failed because the API/settings were absent.
• Real MinIO integration exposed a production defect: 1 failed because MinIO returns an HTTPHeaderDict, not a plain dict, for metadata.
• Same-size server-side corruption: 1 failed because write returned before rereading and digest-validating the stored object.
• Closed operation semantic revalidation: 3 failed because tampered identifiers/flags were not rejected.
• Strict MinIO artifact binding contract: 1 failed because all URI-shaped Parquet refs were previously rejected.
• Repository-attested binding hashes: 1 failed because compiler input rejected the canonical binding_hash.
• Artifact remediation slice: 4 failed, 6 passed before per-plan limits, upload rollback, full schema contracts, and scoped expiry cleanup existed.
• Shared semantic remediation slice: 3 failed, 11 deselected before timezone, expression type replay, and lookup target collision checks existed.
• Runtime remediation slice: 3 failed, 3 deselected before output-binding attestation, exact decimal/timestamptz casts, and separated metrics existed.
• Artifact catalog migration: 1 failed, 6 deselected before migration 20260723_140 existed.
• PostgreSQL artifact resolver: 2 failed, 10 deselected before catalog resolution, re-attestation, and registration existed.
• Dependency SBOM: 1 failed before the CycloneDX JSON document existed.
• Incomplete runtime type contracts: 2 failed, 14 deselected before decimal precision/scale and timestamptz timezone were mandatory.
• Correlation re-attestation: 1 failed, 11 deselected before a catalog row's object key was checked against the requested correlation prefix.

GREEN:

• Final focused unit/schema/SBOM command:
  • PYTHONPATH=. .venv/bin/pytest -q tests/core/data_rules/test_polars_compiler.py tests/runner/test_rule_polars.py tests/runner/test_artifacts.py tests/test_data_rule_schema.py tests/test_data_rule_runtime_sbom.py
  • Result: 42 passed in 1.02s.
• Final real integration command:
  • PYTHONPATH=. .venv/bin/pytest -q tests/integration/test_data_rule_polars_execution.py --show-capture=no -o log_cli=false
  • Result: 1 passed in 1.01s.
• Expanded Task 5 / SQL regression slice during implementation:
  • Result: 197 passed, then 75 passed after repository hardening.

Real integration and cleanup

The integration used runtime values discovered from deploy/docker/docker-compose.yml; credentials were not copied into the test source.

• PostgreSQL: live source at 127.0.0.1:25432.
• MySQL: live lookup source at 127.0.0.1:23306.
• MinIO: live object store at 127.0.0.1:19000.
• Platform PostgreSQL: live canonical plans, bindings, snapshots, and artifact catalog at 127.0.0.1:15432.
• Flow: read PostgreSQL customers, read MySQL segments, write both as bounded Parquet artifacts, register them through PostgresArtifactResolver, load the published fixture through PostgresRulePlanRepository, execute through RulePlanExecutor, normalize → lookup join → assert → deduplicate, register and reread the output artifact.
• Evidence: 4 input rows → 2 output rows, 1 assertion reject, 1 deduplicated row; output values and segment enrichment were verified. A second execution with the same correlation succeeded and retained its identical output digest idempotently in the stable artifact catalog.
• Cleanup: only rules/<test-correlation-id>/ objects were removed. The test asserted the exact prefix was empty afterward. Test-owned source tables and canonical platform rows were also removed.

Full verification

• PYTHONPATH=. .venv/bin/pytest -q
  • 541 passed, 26 skipped, 59 subtests passed in 4.52s.
• docker compose -f deploy/docker/docker-compose.yml config --quiet
  • Passed.
• Ruff over all Task 5 production and test files
  • All checks passed!
• git diff --check
  • Passed.

Dependency and license

• Exact pins: polars==1.42.1, minio==7.2.10.
• Installed into .venv: Polars 1.42.1 and its matching polars-runtime-32==1.42.1.
• Installed metadata: Python >=3.10; MIT license text.
• Added docs/security/data-rule-runtime-sbom.json with versions, licenses, official sources, package URLs, and runtime purposes.
• Plans remain independent of Polars internal serialization formats.

Files

Created:

• app/core/data_rules/compilers/polars.py
• app/runner/artifacts.py
• app/runner/rule_polars.py
• tests/core/data_rules/test_polars_compiler.py
• tests/runner/test_artifacts.py
• tests/runner/test_rule_polars.py
• tests/integration/test_data_rule_polars_execution.py
• migrations/versions/20260723_140_rule_run_artifacts.py
• docs/security/data-rule-runtime-sbom.json
• tests/test_data_rule_runtime_sbom.py

Modified:

• requirements.txt
• app/core/data_rules/compilers/__init__.py
• app/core/data_rules/execution_contracts.py
• app/core/data_rules/repository.py
• app/runner/api.py
• app/runner/bootstrap.py
• app/runner/nodes.py
• app/runner/rules.py
• deploy/docker/docker-compose.yml
• tests/core/data_rules/test_execution_contracts.py
• tests/runner/test_bootstrap.py

Self-review

• Compiler and runtime validators both enforce closed shapes and operation semantics.
• Expressions are reconstructed only from the Task 3 AST and must advertise the polars backend.
• Aggregate functions, lookup join variants, masking policy kinds, regexes, casts, assertion action, identifiers, and resource limits are allowlisted.
• Dataset bindings and lookup contexts are resolved by canonical IDs; repository binding hashes are preserved and re-attested at execution.
• Artifact keys are generated only by the server as rules/<correlation-id>/<artifact-id>.parquet.
• Artifact reads validate store ownership, strict reference shape, content type, size, TTL, digest, row count, schema digest, and memory bounds.
• Artifact reads/writes enforce plan limits before and after transfer, decompression/materialization, and serialization. Full field contracts cover nullability, exact decimal precision/scale, and timestamptz timezone.
• Artifact writes reread the server object before returning a ref, catching same-size corruption, and delete the generated object on validation failure.
• High-risk runtime intermediates are materialized through the same bounded checkpoint; assertion rejects, filters, deduplication, joins, and aggregation have separate counters.
• Artifact catalog resolution is exact-correlation scoped, current binding hashes are re-attested, and expired cleanup is limited to the configured bucket's exact rules/<correlation>/ prefix.
• MinIO credentials stay in Runner settings with repr=False and never enter refs, logs, or plans.

Concerns / deferred work

• Task 7 still owns test-evidence recording and plan publication. RulePlanExecutor correctly refuses compiled plans in the published runtime path.
• quality_check remains fail-closed through the existing adapter; this task did not claim a new artifact-backed quality-check implementation.
• Digest/schema validation requires reading a bounded artifact before exposing a LazyFrame. Lazy operations and Parquet I/O are used, but fully streaming end-to-end execution is intentionally deferred until a streaming digest-verification design preserves the same security checks.