Update examples in README, Add dvf with Variant functions preinstalled by alamb · Pull Request #102 · datafusion-contrib/datafusion-variant

alamb · 2026-10-26T20:34:46Z

Why

part of Add more examples to the main readme #100

Note this PR looks huge but more than half of it is Cargo.lock

I want to make this repo as simple as possible to use / test, and part of that is lowering the friction to ad-hoc testing

Here is my proposal: https://github.com/datafusion-contrib/datafusion-variant/blob/alamb/dfv/README.md

There is a "cli" example, but it:

requires running download.sh to download 100mb of data
It also reinvents some non trivial part of datafusion-cli

Instead, I think it would be better to have a version of datafusion-cli with datafusion-variant pre-integrated so people could test how it works more easily and so that we can test and improve the integration going forward

What

Updates the README to focus on starting up and trying out Variant quickly
Adds a the dfv ("DataFusion Variant") binary that is datafusion-cli with the Variant functions preinstalled

Test it out:

andrewlamb@Andrews-MacBook-Pro-3:~/Software/datafusion-variant$ cargo run
   Compiling dfv v0.1.0 (/Users/andrewlamb/Software/datafusion-variant/dfv)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `dfv` (bin "dfv") generated 1 warning
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2.47s
     Running `target/debug/dfv`
dfv (DataFusion Variant CLI) v0.1.0 based on DataFusion CLI v55.1.0

Try:
  -- Build a table of Variant values and query it
  CREATE TABLE events AS
  SELECT parse_json(json_data) AS event
  FROM (VALUES
      ('{"service":"api","status":200,"latency_ms":12,"tags":["cached"]}'),
      ('{"service":"api","status":500,"latency_ms":85,"error":null}'),
      ('{"service":"api","status":504,"error":"Somethings not working"}')
  ) AS input(json_data);

  -- Select fields from the Variant values using the `:` operator and filter by a field value
  SELECT
     event:service::VARCHAR AS service
  FROM events
  WHERE event:status::BIGINT >= 400;

  -- Write Variant to Parquet and read it back
  COPY events TO 'events.parquet' STORED AS PARQUET;
  SET datafusion.execution.parquet.skip_metadata = false;
  SELECT to_json(event) FROM 'events.parquet';

  -- Inspect the schema of Variant values
  SELECT schema_of_variant(event) FROM events;

See https://github.com/datafusion-contrib/datafusion-variant#functions for all functions.

Enter \q to exit.

>  \? for help, \q to quit.

Planned follow ons:

Remove the examples/cli.rs interactive CLI example #108
Add a load_bsky(n) table function to dfv #109
Consolidate the example program

Add a `dfv` workspace crate whose binary is datafusion-cli's `main.rs` from the pinned DataFusion revision plus `register_all` and the `VariantTypePlanner`. The root workspace lists `dfv` in `default-members` so `cargo run --bin dfv` and `cargo test` work from the repository root. End-to-end tests run the binary with `-c` to check the banner, JSON parsing and field extraction, `:` field access, the SQL `VARIANT` type, a Parquet write/read round trip, and error reporting. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Drop the datafusion-cli options other than `-c` and `-f` (memory and disk limits, Spark mode, output format, rc files, etc.) and the dependencies they needed. Rewrite the end-to-end tests as `dfv/tests/cli_integration.rs` using `assert_cmd` and `predicates`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The dfv REPL test pipes SQL on stdin, and datafusion-cli writes its command history to the working directory. Remove the accidentally committed file and ignore it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

alamb · 2026-10-28T14:47:48Z

 > [!WARNING]
-> `register_all` replaces existing functions with the same names. Individual UDFs
-> remain available for selective registration.
+> `register_all` replaces existing functions with the same names. 


I tried to rewrite the intro to be more succinct and show how to do common tasks.

alamb · 2026-10-28T14:48:17Z

 +---------+------------+
 ```

-## What is Variant?


Moved to the top and make it more concise

alamb · 2026-10-28T14:52:34Z

+
+This repository also includes a `dfv` command-line tool with the Variant
+functions and SQL type preinstalled for you to try the examples interactively. Run it like
+```shell


My goal was to make it super easy to try this out cargo run

alamb · 2026-10-28T14:53:32Z

-
-`parse_json` converts JSON text to the
-[Parquet Variant encoding](https://parquet.apache.org/docs/file-format/types/variantencoding/);
-`CAST(value AS VARIANT)` converts typed scalars and arrays, including decimals;


I am afraid this cast stuff got lost. I can add examples as a follow on PR perhaps

alamb · 2026-10-28T14:53:53Z

-| Objects | [`variant_object_construct`](https://github.com/datafusion-contrib/datafusion-variant/blob/main/tests/test_files/variant_object_construct.slt), [`variant_object_keys`](https://github.com/datafusion-contrib/datafusion-variant/blob/main/tests/test_files/variant_object_keys.slt), [`variant_object_insert`](https://github.com/datafusion-contrib/datafusion-variant/blob/main/tests/test_files/variant_object_insert.slt), [`variant_object_delete`](https://github.com/datafusion-contrib/datafusion-variant/blob/main/tests/test_files/variant_object_delete.slt) |
-| Arrays | [`variant_list_construct`](https://github.com/datafusion-contrib/datafusion-variant/blob/main/tests/test_files/variant_list_construct.slt), [`variant_list_insert`](https://github.com/datafusion-contrib/datafusion-variant/blob/main/tests/test_files/variant_list_insert.slt), [`variant_list_delete`](https://github.com/datafusion-contrib/datafusion-variant/blob/main/tests/test_files/variant_list_delete.slt) |
-| Utilities | [`variant_normalize`](https://github.com/datafusion-contrib/datafusion-variant/blob/main/src/variant_normalize.rs), [`variant_pretty`](https://github.com/datafusion-contrib/datafusion-variant/blob/main/tests/test_files/variant_pretty.slt) |
+| JSON conversion | [`parse_json`, `try_parse_json`, `to_json`][parse-json-slt], [`variant_get_json`][variant-get-json-slt] |


I found out of line links much easier to understand

alamb · 2026-10-28T15:35:43Z

What do you think about this @sdf-jkl ?

Resolve README conflicts: keep the new structure, take the updated `register_all` warning from main, and note that `register_all` now installs the type planner. Drop the explicit type planner from dfv. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

sdf-jkl

👍

Thanks @alamb looks better.

Reading and writing didn't work on my end without parquet = { version = "60", features = ["variant_experimental"] }

sdf-jkl · 2026-10-28T15:49:25Z

 > statements. Individual UDFs remain available for selective registration.

-### 3. Create the events table
+Congratulations -- you can now query semi-structured data with the Variant type!


sdf-jkl · 2026-10-28T15:51:06Z


 ```sql
 SELECT event:service::VARCHAR AS service,
       event:latency_ms::BIGINT AS latency_ms


~~use UDF syntax for one of the options?~~

Saw below

Co-authored-by: Kosta Tarasov <33369833+sdf-jkl@users.noreply.github.com>

alamb · 2026-10-28T16:03:50Z

I think I made the requested changes

sdf-jkl · 2026-10-28T16:11:29Z

🚀

alamb · 2026-10-28T16:58:45Z

Thank you @sdf-jkl -- I also have a bunch of tickets I am working to file

sdf-jkl · 2026-10-28T17:14:45Z

Thanks @alamb 🙏 Really appreciate your help pushing this forward

The interactive CLI example is superseded by dfv (#102), a datafusion-cli with the Variant functions preinstalled. Drop the example, its rustyline dev-dependency, and the README instructions for it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Remove the examples/cli.rs interactive CLI example The interactive CLI example is superseded by dfv (#102), a datafusion-cli with the Variant functions preinstalled. Drop the example, its rustyline dev-dependency, and the README instructions for it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Remove download_data.sh and remaining references to the CLI example The jsonbench prepare command downloads the same Bluesky files itself, so the script is redundant. Point profile_parse_json at that command instead. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Revert "Remove download_data.sh and remaining references to the CLI example" This reverts commit dc10c77. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

alamb · 2026-10-28T21:34:12Z

Thanks @alamb 🙏 Really appreciate your help pushing this forward

No problem -- sorry it took so long. This has been on my list for a while but I think I was waiting for a willing partner to help drive it. Thank you for helping. This is going to be great

alamb changed the title ~~Add dfv: datafusion-cli with Variant functions preinstalled~~ Add dfv: datafusion-cli with Variant functions preinstalled, remove example cli.rs Oct 26, 2026

alamb and others added 3 commits October 28, 2026 09:22

Print example Variant commands when dfv starts its interactive shell

e23fb95

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

alamb changed the title ~~Add dfv: datafusion-cli with Variant functions preinstalled, remove example cli.rs~~ Add dfv: datafusion-cli with Variant functions preinstalled Oct 28, 2026

alamb mentioned this pull request Oct 28, 2026

Remove the examples/cli.rs interactive CLI example #108

Merged

alamb force-pushed the alamb/dfv branch from 7d36936 to 197a4f7 Compare October 28, 2026 14:16

alamb changed the title ~~Add dfv: datafusion-cli with Variant functions preinstalled~~ Update examples in README, Add dff with Variant functions preinstalled Oct 28, 2026

alamb changed the title ~~Update examples in README, Add dff with Variant functions preinstalled~~ Update examples in README, Add dvf with Variant functions preinstalled Oct 28, 2026

alamb and others added 3 commits October 28, 2026 10:20

Ignore .history written by the datafusion-cli shell

605c892

The dfv REPL test pipes SQL on stdin, and datafusion-cli writes its command history to the working directory. Remove the accidentally committed file and ignore it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Update README

cd60f5d

Update readme

f50021d

alamb force-pushed the alamb/dfv branch from 197a4f7 to f50021d Compare October 28, 2026 14:21

alamb mentioned this pull request Oct 28, 2026

Add a load_bsky(n) table function to dfv #109

Merged

alamb added 2 commits October 28, 2026 10:46

Clean up

6ed62f9

remove cruft

04647e1

alamb commented Oct 28, 2026

View reviewed changes

alamb mentioned this pull request Oct 28, 2026

register_all should also install the SQL VARIANT type #110

Closed

alamb marked this pull request as ready for review October 28, 2026 15:35

alamb and others added 2 commits October 28, 2026 11:41

updates

e8f4245

sdf-jkl requested changes Oct 28, 2026

View reviewed changes

Apply suggestion from @sdf-jkl

c5cd23e

Co-authored-by: Kosta Tarasov <33369833+sdf-jkl@users.noreply.github.com>

sdf-jkl merged commit 1c32619 into main Oct 28, 2026
3 checks passed

sdf-jkl deleted the alamb/dfv branch October 28, 2026 16:11

sdf-jkl mentioned this pull request Oct 28, 2026

Epic: Apache Spark Variant compatibility #69

Open

12 of 20 tasks

Conversation

alamb commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What

Planned follow ons:

Uh oh!

alamb Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alamb Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

alamb commented Oct 28, 2026

Uh oh!

sdf-jkl left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sdf-jkl Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

sdf-jkl Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alamb commented Oct 28, 2026

Uh oh!

Uh oh!

sdf-jkl commented Oct 28, 2026

Uh oh!

alamb commented Oct 28, 2026

Uh oh!

sdf-jkl commented Oct 28, 2026

Uh oh!

alamb commented Oct 28, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

alamb commented Oct 26, 2026 •

edited

Loading

sdf-jkl left a comment •

edited

Loading