Skip to content

Instantiate stateful scalar UDFs per physical expression - #158

Merged
osipovartem merged 1 commit into
df55-bound-null-object-valuesfrom
df55-physical-udf-instance
Oct 21, 2026
Merged

osipovartem merged 1 commit into
df55-bound-null-object-valuesfrom
df55-physical-udf-instance

Conversation

@osipovartem

@osipovartem osipovartem commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Add an opt-in local scalar UDF hook that creates one invocation object when a physical scalar expression is built.
  • Keep the registered UDF for metadata/coercion while invoking the physical instance for batch execution.
  • Preserve the instance across physical-expression rewrites and forward the hook through aliases.

Motivation

Rustice SEQ functions need state shared across Arrow batches in one query but reset for the next query or view execution. A logical UDF instance can survive in saved plans; batch-local state resets too early.

Validation

  • datafusion-physical-expr: 1610 passed, 2 ignored (full library suite).
  • Focused physical-instance and alias tests pass.
  • cargo fmt --check, clippy -p datafusion-physical-expr --lib --locked, and git diff --check pass.
  • Independent read-only review: approved, no blocking findings.

Scope

This is intentionally a fork-local API. The stable-ABI FFI UDF wrapper does not forward this new hook; plan clones share the same invocation object. Ordinary UDFs retain their existing Arc and do not pay an extra per-batch branch.

CI context

The fork CI currently has unrelated failures outside this three-file diff: CLI integration cannot pull its MinIO image; an existing CSV diagnostic test compares a backtrace-bearing error string; Taplo reports the unchanged datafusion/datasource/Cargo.toml; generated function docs are stale on the base branch. The targeted datafusion-physical-expr suite (1610 passed, 2 ignored), local Rustice integration suite (3850/3850), formatting, and Clippy checks passed. The fork-only FFI limitation remains documented above.

@osipovartem
osipovartem merged commit f6ee532 into df55-bound-null-object-values Oct 21, 2026
33 of 38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant