# Task 5 Report — Polars cross-source compilation and artifact I/O

Date: 2026-07-23
Branch: `codex/data-rule-execution-m3a-m5`
Lifecycle boundary: physical Polars plans are persisted as `compiled`; this task does not publish them.

## Outcome

Implemented a closed JSON Polars compiler, allowlisted LazyFrame reconstruction, server-owned digest-bound Parquet artifact storage, PostgreSQL-backed artifact binding resolution, Runner registration, canonical plan attestations, and a real PostgreSQL + MySQL + MinIO integration.

The plan contains canonical RuleVersion, SchemaSnapshot, and DatasetBinding hashes plus `dataops-polars-1.42.1` provenance. It contains no Python source, pickle, callable, module, client path, arbitrary URL, secret, or Polars internal serialized plan.

## TDD evidence

RED evidence was captured before each production slice:

- Compiler module: `5 failed` because `app.core.data_rules.compilers.polars` did not exist.
- ArtifactStore and adapter modules: `8 failed` because the modules did not exist.
- Cross-source compiler registry/service: `2 failed` because the registry rejected cross-source artifacts.
- Runner canonical attestation/correlation: `1 failed` because correlation was not forwarded.
- Node registry trusted context: `1 failed` because `correlation_id` was not accepted.
- Artifact describe/bootstrap configuration: `2 failed` because the API/settings were absent.
- Real MinIO integration exposed a production defect: `1 failed` because MinIO returns an `HTTPHeaderDict`, not a plain `dict`, for metadata.
- Same-size server-side corruption: `1 failed` because write returned before rereading and digest-validating the stored object.
- Closed operation semantic revalidation: `3 failed` because tampered identifiers/flags were not rejected.
- Strict MinIO artifact binding contract: `1 failed` because all URI-shaped Parquet refs were previously rejected.
- Repository-attested binding hashes: `1 failed` because compiler input rejected the canonical `binding_hash`.

GREEN:

- Final required focused command:
  - `PYTHONPATH=. .venv/bin/pytest -q tests/core/data_rules/test_polars_compiler.py tests/runner/test_rule_polars.py tests/runner/test_artifacts.py tests/integration/test_data_rule_polars_execution.py`
  - Result: `23 passed in 0.95s`.
- Expanded Task 5 / SQL regression slice during implementation:
  - Result: `197 passed`, then `75 passed` after repository hardening.

## Real integration and cleanup

The integration used runtime values discovered from `deploy/docker/docker-compose.yml`; credentials were not copied into the test source.

- PostgreSQL: live source at `127.0.0.1:25432`.
- MySQL: live lookup source at `127.0.0.1:23306`.
- MinIO: live object store at `127.0.0.1:19000`.
- Flow: read PostgreSQL customers, read MySQL segments, write both as bounded Parquet artifacts, compile and execute normalize → lookup join → assert → deduplicate, write and reread the output artifact.
- Evidence: 4 input rows → 2 output rows, 2 rejected rows, 1 assertion violation; output values and segment enrichment were verified.
- Cleanup: only `rules/<test-correlation-id>/` objects were removed. The test asserted the exact prefix was empty afterward. Test-owned PostgreSQL/MySQL tables were also dropped.

## Full verification

- `PYTHONPATH=. .venv/bin/pytest -q`
  - `527 passed, 26 skipped, 59 subtests passed in 4.69s`.
- `docker compose -f deploy/docker/docker-compose.yml config --quiet`
  - Passed.
- Ruff over all Task 5 production and test files
  - `All checks passed!`
- `git diff --check`
  - Passed.

## Dependency and license

- Added exact pin: `polars==1.42.1`.
- Installed into `.venv`: Polars `1.42.1` and its matching `polars-runtime-32==1.42.1`.
- Installed metadata: Python `>=3.10`; MIT license text.
- Plans remain independent of Polars internal serialization formats.

## Files

Created:

- `app/core/data_rules/compilers/polars.py`
- `app/runner/artifacts.py`
- `app/runner/rule_polars.py`
- `tests/core/data_rules/test_polars_compiler.py`
- `tests/runner/test_artifacts.py`
- `tests/runner/test_rule_polars.py`
- `tests/integration/test_data_rule_polars_execution.py`

Modified:

- `requirements.txt`
- `app/core/data_rules/compilers/__init__.py`
- `app/core/data_rules/execution_contracts.py`
- `app/core/data_rules/repository.py`
- `app/runner/api.py`
- `app/runner/bootstrap.py`
- `app/runner/nodes.py`
- `app/runner/rules.py`
- `deploy/docker/docker-compose.yml`
- `tests/core/data_rules/test_execution_contracts.py`
- `tests/runner/test_bootstrap.py`

## Self-review

- Compiler and runtime validators both enforce closed shapes and operation semantics.
- Expressions are reconstructed only from the Task 3 AST and must advertise the `polars` backend.
- Aggregate functions, lookup join variants, masking policy kinds, regexes, casts, assertion action, identifiers, and resource limits are allowlisted.
- Dataset bindings and lookup contexts are resolved by canonical IDs; repository binding hashes are preserved and re-attested at execution.
- Artifact keys are generated only by the server as `rules/<correlation-id>/<artifact-id>.parquet`.
- Artifact reads validate store ownership, strict reference shape, content type, size, TTL, digest, row count, schema digest, and memory bounds.
- Artifact writes reread the server object before returning a ref, catching same-size corruption.
- MinIO credentials stay in Runner settings with `repr=False` and never enter refs, logs, or plans.

## Concerns / deferred work

- Task 7 still owns test-evidence recording and plan publication. `RulePlanExecutor` correctly refuses `compiled` plans in the published runtime path.
- `quality_check` remains fail-closed through the existing adapter; this task did not claim a new artifact-backed quality-check implementation.
- Digest/schema validation requires reading a bounded artifact before exposing a LazyFrame. Lazy operations and Parquet I/O are used, but fully streaming end-to-end execution is intentionally deferred until a streaming digest-verification design preserves the same security checks.