Establish Spark Variant contract coverage #70

Part of #69. Establish executable contracts against Apache Spark, including newer releases and development APIs, before implementing the compatibility fixes in #71. Follow APIs and behavior merged into Spark's default branch; neither 4.2 nor 4.3 is a ceiling. Spark 4.2.0 remains historical test provenance.

Baseline and sources

Preserve the original Spark 4.2.0 baseline at 32f7299601108917fb01920a54e084595b7b3bf8. For additions and changed contracts, record the exact upstream commit and source/test references per function, along with ANSI mode, timezone, and duplicate-key settings. Distinguish implemented upstream APIs from open proposals; refresh proposal expectations when upstream settles the contract.

Use Spark's Variant SQL tests, expected results, VariantExpressionSuite, and VariantSuite. Extend #68, reuse audit cases, and use equivalent typed-getter, extraction-plus-cast, and try-getter paths. Separate typed getters were removed in #86; #67 is closed. Preserve upstream attribution and license notices.

Coverage already landed

Implementation and test inventory refreshed against 95cb044 (Arrow 60) on 2026-10-28. Completed entries mean the implementation or coverage landed; they do not assert full Spark parity. Spark provenance remains pinned separately in each suite.

As of #111, register_all installs the SQL VARIANT type planner. The SLT harness no longer installs it separately, so the existing SQL-type cases cover registration through register_all.

register_all also installs the explode planners (#117) and Variant formatter (#118); the SLT and CLI suites exercise those registration paths.

These suites include explicit skips for remaining parser, cast, path, and shredded-input gaps. The source comments pin different Spark commits for different contracts; retain that provenance instead of relabeling old expectations as results from a newer Spark release. Some expectations were executed on Spark and others were checked against source/tests; preserve that distinction.

The validation suite pins Spark at 0aa4f9136bccc40856a5af688e3d16f8b38ebfc4; expectations were checked against its source/tests, not by running Spark. Active cases record accepted differences: Arrow rejects malformed UTF-8 and unused malformed metadata that Spark accepts, and accepts the time primitive that this Spark revision rejects. The malformed shredded-metadata panic remains a defect tracked by apache/arrow-rs#11069.

The #105 conversion suite adapts Spark's cast to variant/to_variant_object unit cases at 32f7299601108917fb01920a54e084595b7b3bf8, sets UTC, and records formatting adaptations. Active cases cover scalar/container conversion, planning-time type rejection, extension metadata, and per-row TRY_CAST failures. Skips retain pending expectations for nested Variant inputs, null object fields, and duplicate keys. The #107 additions exercise local SQL dispatch; they inherit getter-kernel limitations, including a skipped shredded Variant-null-to-struct case.

The #117 explode suite pins Spark at 653db2957ac7d1b18d8f0b070fe8b3b01ba34c9d; expectations were checked against source/tests, not by running Spark. It covers arrays/objects, empty/scalar/null input, child Variant nulls, nested calls, aliases, and named arguments. Storage-layout, schema, and planner regressions also exercise local integration. Correlated LEFT joins remain unsupported, result nullability is conservative, and Arrow validates nested payloads more eagerly than Spark; dependencies are tracked in #71. Active error cases for these limitations do not establish Spark parity.

Coverage to complete and extend

• Functions: complete coverage for the implemented functions above, including remaining parser/getter/serializer gaps. Both explode functions, is_valid_variant, and to_variant_object now have active suites. General Spark functions that also accept VARIANT (from_json(..., 'VARIANT'), Support Spark from_json(json, 'variant') #140; typeof) are out of scope.
• Newer APIs: cover variant_insert / try_variant_insert, variant_set / try_variant_set, variant_array_append / try_variant_array_append, variant_delete, variant_strip_nulls, variant_pick, and variant_from_arrays / variant_from_entries. Use the upstream inventory and implementation-reuse mapping in Close the function compatibility gaps with Apache Spark #71. Track variant_array_length, recursive explode, and internal Variant canonicalization (Close the function compatibility gaps with Apache Spark #71 / SPARK-59151) separately as proposals until their contracts land. Include duplicate keys, nested mutation, ordered deletion, overlapping selection paths, array compaction/padding, null handling, and strict/try error boundaries.
• Path existence: track future VARIANT input to Spark's JSON_EXISTS under Close the function compatibility gaps with Apache Spark #71 / SPARK-58687. Its landed implementation accepts STRING; VARIANT input is deferred. Pin present-null versus missing, SQL NULL, lax-path behavior, and error handling before adding support.
• Paths and casts: root/nested/indexed/quoted/escaped and malformed paths; SQL type hints; strict/try behavior; numeric, string, decimal, and temporal conversions, including precision, scale, rounding, and overflow.
• Nulls and arguments: SQL NULL, Variant null, missing paths, scalar/column inputs, literal/varying/null paths, empty batches, and invalid types.
• Parsing and serialization: malformed JSON, duplicate keys, large integers, precise decimals, special floats, null object members, formatting, and options such as timezone.
• Schemas and explode: extend inferred-type/merge coverage and close the remaining explode nullability, correlated LEFT-join, and malformed-input contract gaps above.
• SQL integration: extend the landed :, getter casts, SQL VARIANT, and casts to/from Variant with remaining syntax, nested-target, and type-resolution/coercion cases; link planner dependencies to Epic: Apache Spark Variant compatibility #69.
• Arrow boundary: extension metadata, exact result types/nullability, and equivalent shredded/unshredded results. Reuse upstream fixtures; Arrow owns encoding and shredding correctness.

Test requirements

Use SLTs for SQL-visible contracts, supplying Rust fixtures when SQL cannot construct malformed storage or metadata. Keep Rust unit tests for invariants that SLTs cannot reliably exercise, such as empty batches or exact decimal storage; avoid duplicating Arrow's decoder coverage. Check meaningful errors; panics are failures, and exact Spark diagnostic wording is unnecessary.

Keep original Spark calls and adaptations visible so renamed functions, paths, or type hints cannot hide gaps. For agreed Spark behavior, preserve unsupported expectations as explicit skips with reasons and linked fixes. Accepted specification- and contract-compliant deviations need active tests, the Spark result, and a rationale.

Maintain a small coverage index linking contracts, tests, sources, and status: matching, accepted deviation, blocked, or out of scope with a reason. Account for every audit gap. Distinguish crate-specific helpers; API retention is a separate decision.

Completion

• Land the initial Spark-derived suites listed above.
• Complete coverage for missing functions and newer APIs, and check in a maintained coverage index (the issue table above is only an initial inventory).
• Run supported cases in existing tests/CI and verify skips preserve pending expectations.
• Document how to validate and refresh expectations against pinned Spark, including settings and adaptations.

Normal tests/CI must not require Spark; a permanent cross-engine runner is out of scope. This issue establishes contracts; implementing parity and enabling pending cases belongs to #71.