# Task 5 Report — Polars cross-source compilation and artifact I/O Date: 2026-07-23 Branch: `codex/data-rule-execution-m3a-m5` Lifecycle boundary: physical Polars plans are created as `compiled`; this task does not add a production promotion API. The real execution test inserts a published fixture only to exercise the pre-existing Runner publication gate. ## Outcome Implemented a closed JSON Polars compiler, shared compile/runtime semantic validation, allowlisted LazyFrame reconstruction, server-owned digest-bound Parquet artifact storage, correlation-scoped PostgreSQL artifact handoff, current output-binding re-attestation, separated operation metrics, Runner registration, canonical plan attestations, and a real PostgreSQL + MySQL + MinIO integration through the production repository/resolver/executor path. The plan contains canonical RuleVersion, SchemaSnapshot, and DatasetBinding hashes plus `dataops-polars-1.42.1` provenance. It contains no Python source, pickle, callable, module, client path, arbitrary URL, secret, or Polars internal serialized plan. ## TDD evidence RED evidence was captured before each production slice: - Compiler module: `5 failed` because `app.core.data_rules.compilers.polars` did not exist. - ArtifactStore and adapter modules: `8 failed` because the modules did not exist. - Cross-source compiler registry/service: `2 failed` because the registry rejected cross-source artifacts. - Runner canonical attestation/correlation: `1 failed` because correlation was not forwarded. - Node registry trusted context: `1 failed` because `correlation_id` was not accepted. - Artifact describe/bootstrap configuration: `2 failed` because the API/settings were absent. - Real MinIO integration exposed a production defect: `1 failed` because MinIO returns an `HTTPHeaderDict`, not a plain `dict`, for metadata. - Same-size server-side corruption: `1 failed` because write returned before rereading and digest-validating the stored object. - Closed operation semantic revalidation: `3 failed` because tampered identifiers/flags were not rejected. - Strict MinIO artifact binding contract: `1 failed` because all URI-shaped Parquet refs were previously rejected. - Repository-attested binding hashes: `1 failed` because compiler input rejected the canonical `binding_hash`. - Artifact remediation slice: `4 failed, 6 passed` before per-plan limits, upload rollback, full schema contracts, and scoped expiry cleanup existed. - Shared semantic remediation slice: `3 failed, 11 deselected` before timezone, expression type replay, and lookup target collision checks existed. - Runtime remediation slice: `3 failed, 3 deselected` before output-binding attestation, exact decimal/timestamptz casts, and separated metrics existed. - Artifact catalog migration: `1 failed, 6 deselected` before migration `20260723_140` existed. - PostgreSQL artifact resolver: `2 failed, 10 deselected` before catalog resolution, re-attestation, and registration existed. - Dependency SBOM: `1 failed` before the CycloneDX JSON document existed. - Incomplete runtime type contracts: `2 failed, 14 deselected` before decimal precision/scale and timestamptz timezone were mandatory. - Correlation re-attestation: `1 failed, 11 deselected` before a catalog row's object key was checked against the requested correlation prefix. GREEN: - Final focused unit/schema/SBOM command: - `PYTHONPATH=. .venv/bin/pytest -q tests/core/data_rules/test_polars_compiler.py tests/runner/test_rule_polars.py tests/runner/test_artifacts.py tests/test_data_rule_schema.py tests/test_data_rule_runtime_sbom.py` - Result: `42 passed in 1.02s`. - Final real integration command: - `PYTHONPATH=. .venv/bin/pytest -q tests/integration/test_data_rule_polars_execution.py --show-capture=no -o log_cli=false` - Result: `1 passed in 1.01s`. - Expanded Task 5 / SQL regression slice during implementation: - Result: `197 passed`, then `75 passed` after repository hardening. ## Real integration and cleanup The integration used runtime values discovered from `deploy/docker/docker-compose.yml`; credentials were not copied into the test source. - PostgreSQL: live source at `127.0.0.1:25432`. - MySQL: live lookup source at `127.0.0.1:23306`. - MinIO: live object store at `127.0.0.1:19000`. - Platform PostgreSQL: live canonical plans, bindings, snapshots, and artifact catalog at `127.0.0.1:15432`. - Flow: read PostgreSQL customers, read MySQL segments, write both as bounded Parquet artifacts, register them through `PostgresArtifactResolver`, load the published fixture through `PostgresRulePlanRepository`, execute through `RulePlanExecutor`, normalize → lookup join → assert → deduplicate, register and reread the output artifact. - Evidence: 4 input rows → 2 output rows, 1 assertion reject, 1 deduplicated row; output values and segment enrichment were verified. A second execution with the same correlation succeeded and retained its identical output digest idempotently in the stable artifact catalog. - Cleanup: only `rules//` objects were removed. The test asserted the exact prefix was empty afterward. Test-owned source tables and canonical platform rows were also removed. ## Full verification - `PYTHONPATH=. .venv/bin/pytest -q` - `541 passed, 26 skipped, 59 subtests passed in 4.52s`. - `docker compose -f deploy/docker/docker-compose.yml config --quiet` - Passed. - Ruff over all Task 5 production and test files - `All checks passed!` - `git diff --check` - Passed. ## Dependency and license - Exact pins: `polars==1.42.1`, `minio==7.2.10`. - Installed into `.venv`: Polars `1.42.1` and its matching `polars-runtime-32==1.42.1`. - Installed metadata: Python `>=3.10`; MIT license text. - Added `docs/security/data-rule-runtime-sbom.json` with versions, licenses, official sources, package URLs, and runtime purposes. - Plans remain independent of Polars internal serialization formats. ## Files Created: - `app/core/data_rules/compilers/polars.py` - `app/runner/artifacts.py` - `app/runner/rule_polars.py` - `tests/core/data_rules/test_polars_compiler.py` - `tests/runner/test_artifacts.py` - `tests/runner/test_rule_polars.py` - `tests/integration/test_data_rule_polars_execution.py` - `migrations/versions/20260723_140_rule_run_artifacts.py` - `docs/security/data-rule-runtime-sbom.json` - `tests/test_data_rule_runtime_sbom.py` Modified: - `requirements.txt` - `app/core/data_rules/compilers/__init__.py` - `app/core/data_rules/execution_contracts.py` - `app/core/data_rules/repository.py` - `app/runner/api.py` - `app/runner/bootstrap.py` - `app/runner/nodes.py` - `app/runner/rules.py` - `deploy/docker/docker-compose.yml` - `tests/core/data_rules/test_execution_contracts.py` - `tests/runner/test_bootstrap.py` ## Self-review - Compiler and runtime validators both enforce closed shapes and operation semantics. - Expressions are reconstructed only from the Task 3 AST and must advertise the `polars` backend. - Aggregate functions, lookup join variants, masking policy kinds, regexes, casts, assertion action, identifiers, and resource limits are allowlisted. - Dataset bindings and lookup contexts are resolved by canonical IDs; repository binding hashes are preserved and re-attested at execution. - Artifact keys are generated only by the server as `rules//.parquet`. - Artifact reads validate store ownership, strict reference shape, content type, size, TTL, digest, row count, schema digest, and memory bounds. - Artifact reads/writes enforce plan limits before and after transfer, decompression/materialization, and serialization. Full field contracts cover nullability, exact decimal precision/scale, and timestamptz timezone. - Artifact writes reread the server object before returning a ref, catching same-size corruption, and delete the generated object on validation failure. - High-risk runtime intermediates are materialized through the same bounded checkpoint; assertion rejects, filters, deduplication, joins, and aggregation have separate counters. - Artifact catalog resolution is exact-correlation scoped, current binding hashes are re-attested, and expired cleanup is limited to the configured bucket's exact `rules//` prefix. - MinIO credentials stay in Runner settings with `repr=False` and never enter refs, logs, or plans. ## Concerns / deferred work - Task 7 still owns test-evidence recording and plan publication. `RulePlanExecutor` correctly refuses `compiled` plans in the published runtime path. - `quality_check` remains fail-closed through the existing adapter; this task did not claim a new artifact-backed quality-check implementation. - Digest/schema validation requires reading a bounded artifact before exposing a LazyFrame. Lazy operations and Parquet I/O are used, but fully streaming end-to-end execution is intentionally deferred until a streaming digest-verification design preserves the same security checks.