DataOps-platform/.superpowers/sdd/task-5-report.md

Task 5 Report — Polars cross-source compilation and artifact I/O

Date: 2026-07-23 Branch: codex/data-rule-execution-m3a-m5 Lifecycle boundary: physical Polars plans are persisted as compiled; this task does not publish them.

Outcome

Implemented a closed JSON Polars compiler, allowlisted LazyFrame reconstruction, server-owned digest-bound Parquet artifact storage, PostgreSQL-backed artifact binding resolution, Runner registration, canonical plan attestations, and a real PostgreSQL + MySQL + MinIO integration.

The plan contains canonical RuleVersion, SchemaSnapshot, and DatasetBinding hashes plus dataops-polars-1.42.1 provenance. It contains no Python source, pickle, callable, module, client path, arbitrary URL, secret, or Polars internal serialized plan.

TDD evidence

RED evidence was captured before each production slice:

• Compiler module: 5 failed because app.core.data_rules.compilers.polars did not exist.
• ArtifactStore and adapter modules: 8 failed because the modules did not exist.
• Cross-source compiler registry/service: 2 failed because the registry rejected cross-source artifacts.
• Runner canonical attestation/correlation: 1 failed because correlation was not forwarded.
• Node registry trusted context: 1 failed because correlation_id was not accepted.
• Artifact describe/bootstrap configuration: 2 failed because the API/settings were absent.
• Real MinIO integration exposed a production defect: 1 failed because MinIO returns an HTTPHeaderDict, not a plain dict, for metadata.
• Same-size server-side corruption: 1 failed because write returned before rereading and digest-validating the stored object.
• Closed operation semantic revalidation: 3 failed because tampered identifiers/flags were not rejected.
• Strict MinIO artifact binding contract: 1 failed because all URI-shaped Parquet refs were previously rejected.
• Repository-attested binding hashes: 1 failed because compiler input rejected the canonical binding_hash.

GREEN:

• Final required focused command:
  • PYTHONPATH=. .venv/bin/pytest -q tests/core/data_rules/test_polars_compiler.py tests/runner/test_rule_polars.py tests/runner/test_artifacts.py tests/integration/test_data_rule_polars_execution.py
  • Result: 23 passed in 0.95s.
• Expanded Task 5 / SQL regression slice during implementation:
  • Result: 197 passed, then 75 passed after repository hardening.

Real integration and cleanup

The integration used runtime values discovered from deploy/docker/docker-compose.yml; credentials were not copied into the test source.

• PostgreSQL: live source at 127.0.0.1:25432.
• MySQL: live lookup source at 127.0.0.1:23306.
• MinIO: live object store at 127.0.0.1:19000.
• Flow: read PostgreSQL customers, read MySQL segments, write both as bounded Parquet artifacts, compile and execute normalize → lookup join → assert → deduplicate, write and reread the output artifact.
• Evidence: 4 input rows → 2 output rows, 2 rejected rows, 1 assertion violation; output values and segment enrichment were verified.
• Cleanup: only rules/<test-correlation-id>/ objects were removed. The test asserted the exact prefix was empty afterward. Test-owned PostgreSQL/MySQL tables were also dropped.

Full verification

• PYTHONPATH=. .venv/bin/pytest -q
  • 527 passed, 26 skipped, 59 subtests passed in 4.69s.
• docker compose -f deploy/docker/docker-compose.yml config --quiet
  • Passed.
• Ruff over all Task 5 production and test files
  • All checks passed!
• git diff --check
  • Passed.

Dependency and license

• Added exact pin: polars==1.42.1.
• Installed into .venv: Polars 1.42.1 and its matching polars-runtime-32==1.42.1.
• Installed metadata: Python >=3.10; MIT license text.
• Plans remain independent of Polars internal serialization formats.

Files

Created:

• app/core/data_rules/compilers/polars.py
• app/runner/artifacts.py
• app/runner/rule_polars.py
• tests/core/data_rules/test_polars_compiler.py
• tests/runner/test_artifacts.py
• tests/runner/test_rule_polars.py
• tests/integration/test_data_rule_polars_execution.py

Modified:

• requirements.txt
• app/core/data_rules/compilers/__init__.py
• app/core/data_rules/execution_contracts.py
• app/core/data_rules/repository.py
• app/runner/api.py
• app/runner/bootstrap.py
• app/runner/nodes.py
• app/runner/rules.py
• deploy/docker/docker-compose.yml
• tests/core/data_rules/test_execution_contracts.py
• tests/runner/test_bootstrap.py

Self-review

• Compiler and runtime validators both enforce closed shapes and operation semantics.
• Expressions are reconstructed only from the Task 3 AST and must advertise the polars backend.
• Aggregate functions, lookup join variants, masking policy kinds, regexes, casts, assertion action, identifiers, and resource limits are allowlisted.
• Dataset bindings and lookup contexts are resolved by canonical IDs; repository binding hashes are preserved and re-attested at execution.
• Artifact keys are generated only by the server as rules/<correlation-id>/<artifact-id>.parquet.
• Artifact reads validate store ownership, strict reference shape, content type, size, TTL, digest, row count, schema digest, and memory bounds.
• Artifact writes reread the server object before returning a ref, catching same-size corruption.
• MinIO credentials stay in Runner settings with repr=False and never enter refs, logs, or plans.

Concerns / deferred work

• Task 7 still owns test-evidence recording and plan publication. RulePlanExecutor correctly refuses compiled plans in the published runtime path.
• quality_check remains fail-closed through the existing adapter; this task did not claim a new artifact-backed quality-check implementation.
• Digest/schema validation requires reading a bounded artifact before exposing a LazyFrame. Lazy operations and Parquet I/O are used, but fully streaming end-to-end execution is intentionally deferred until a streaming digest-verification design preserves the same security checks.