Zoi + Ecto and Changesets

[phcurado]

Following the discussion on Elixir Forum, we should have a way to create Zoi schemas and generate different outputs such as ecto embedded schemas. Ecto will be an optional library, and if the project have it, Zoi will offer functions that will help integrating ecto schemas (for now embeds) with zoi plain schemas.

So far the API design agreed was:

defmodule Project.Inputs.OnboardUser do
   require Zoi.Ecto

  @schema Zoi.map(%{email: Zoi.email()})

  # Macro that will create the embedded in this module
  Zoi.Ecto.generate_embedded_schema(@schema)

  def schema, do: @schema
end

Zoi should also provide a way to integrate with changesets. For now we agreed to convert zoi parsed results (or errors) into a changeset container but the API will be later defined

[phcurado]

as per comment in the forum, it would be great to integrate into discriminated_union , extend, pick.
There is also interest to integrate with ecto schemas (and not embeds only) .

Discriminated Union

As for discriminated_union, there is no type equivalent on Ecto. I could store as map and just parse as discriminated_union under the hood.

Extend

extends a map schema with additional fields. For Ecto generation this should just work since the result is still a Zoi.Types.Map with merged fields.

Pick

This is probably to select a subset of fields from a map schema. Same as extend, the result is a Zoi.Types.Map so it should already work.

Omit

Since supporting pick, would be good to support omit as well since it's the opposite

Integration with Ecto Schemas

This might be much more complex since it represents a data on DB and have many possibilities on Ecto API.
Ecto API supports belongs_to, has_one, has_many, etc each one with different opts on how this relation works so I might end up reimplementing the ecto schema api. Good to analyze in another issue/feature

[phcurado]

Created dedicated issue tracker for pick and omit: #160

[dimitarvp]

Good call on making dedicated issue for pick/omit. Helps us maintain focused discussion. 👍🏻

So for my use case, I work on a small collection of financial microservices and apps -- think Ecto, Phoenix, Absinthe, and various REST API parsing/validation bolted on top -- and as mentioned on the EF post, we have the same data entering through multiple edges/boundaries f.ex. a GraphQL mutation, SQS consumer, REST endpoint. And in each place the team has hand-rolled nil checks, Date parsing, and the tricky dragon to slay: conditional validation. We already had real bugs or just wasted work related to this like double or even triple Date parsing, missing required field checks (too much LLM slop), fields written twice or more but with different types.

For the moment I'd not use :map for discriminated_union btw. We need something more akin the old-school DB table polymorphism i.e. you have 20 fields in a table but if :type == "A" then only 13 are relevant.

To give you a concrete code example (with some business details obfuscated), currently we have product_type and we do this:

def validate_onboard_args(:card_type_1, args) do
  missing = [:partner_account_number, :initial_balance, :origination_date]
            |> Enum.filter(&is_nil(args[&1]))
  # ...then separately parse dates, check formats, etc.
end

def validate_onboard_args(:card_type_2, args) do
  if is_nil(args[:last_four]) or args[:last_four] == "", do: {:error, "..."}, else: :ok
end

While I would much prefer to have something like this:

@schema Zoi.discriminated_union("product_type", %{
  card_type_1: Zoi.map(%{
    partner_account_number: Zoi.string(),
    initial_balance: Zoi.integer() |> Zoi.min(0),
    # ISO8601 string -> Date
    origination_date: Zoi.date(coerce: true),
    promo_end_date: Zoi.date(coerce: true) |> Zoi.optional()
  }),
  card_type_2: Zoi.map(%{
    last_four: Zoi.string() |> Zoi.min(4) |> Zoi.max(4)
  })
})

And then just: Zoi.Ecto.generate_embedded_schema(@schema)

⚠️ One thing I omitted for brevity is that I want the different variants of the union to have common fields -- the desired feature is not to have each variant have fields that are exclusively reserved for it. Think billing_address of a card in the above example. Each card's product type should have it attached.

Continuing on the discriminated_union impl note from above: they should just generate a flat embedded schema IMO i.e. each variant's fields should just be optional from the POV of Ecto. Zoi should validate at parse time. And Ecto does not need to know about the discrimination; it's just the struct / Changeset container really. But in the example above product_type must be an Ecto.Enum -- I think this would help your code as well (but do let me know), plus it communicates intent much better.

So Zoi will own the "which fields are required when" via the discriminated union. Ecto just holds the data. This avoids the :map type which would lose field-level error reporting and struct access.

While I was writing this, I remembered two other annoyances that are very adjacent to this bigger request:

• String-keyed input in changeset/2: Phoenix/Absinthe pass %{"email" => "..."} and not %{email: "..."}. I see Zoi handles this in regular parsing but the PR tests only use atom keys. Would be good to test/verify? I might have misread.
• parse/1 convenience if that's OK: would it make sense for generate_embedded_schema to also generate a parse(params) function that returns {:ok, %__MODULE__{}} | {:error, %Ecto.Changeset{}}? That's the API we'd call at every system boundary. I can of course hand-roll this in a literal minute but I don't see why not add a convenience wrapper as a part of this feature request in Zoi.

Currently we do not need full Ecto schema support i.e. belongs_to and has_one / has_many etc. IMO embedded schemas are quite enough for now (or might even end up being enough in general and Zoi will never have to touch the other DB-linked table relationships). Our DB schemas are a separate concern and should be out of scope here -- unless you find it easier to ship both together. That's up to you. For now we only want Zoi at the validation/parsing boundary (and not trying to replace Ecto for persistence, including relationships).

I would love to allocate a little time to test against your PR as it progresses. That's a time/energy cost I am willing to bear until this is done. And we have even more scenarios, this is just the one that's freshest in my head.

If I left anything vague please call me out immediately. I want us to arrive at a spec that's actionable for you.

[phcurado]

amazing description thank you. This gave me a lot of insights.

I pushed discriminated_union into embeds PR if you want to check, I did following the behaviour you described. I didn't add the parse function because for now I don't want to infer the API, I much rather generate the embedded and let the user/client add their functions.

You mentioned somethign about conditional validation, Zoi supports through refinements, let me know if this helps you:

  Zoi.map(%{
    contact_method: Zoi.enum([:email, :phone]),
    email: Zoi.optional(Zoi.string()),
    phone: Zoi.optional(Zoi.string())
  })
  |> Zoi.refine(fn
    %{contact_method: :phone, phone: nil} -> {:error, "phone is required"}
    _ -> :ok
  end)

Refinement docs: https://hexdocs.pm/zoi/Zoi.html#module-refinements

The string/atom keys you mentioned, should already work. Zoi have a lot of tests for this in the Map module.

Also I merged pick and omit to main branch and added to the PR, it's available to use now.

[phcurado]

general question, I re-read your use-case, why do you need embedded schema? I understood we can centralise the definition on Zoi, then we can output to multiple sources like embeds but I didn't understand the core behind using them

[dimitarvp]

First, thanks for DiscriminatedSchema -- looks like what I was hoping for.

RE: embedded Ecto.Schema -- embedded schemas with @primary_key false are the standard Phoenix pattern for boundary validation, more or less. Your bridge automates what people already do by hand, en masse.

Thinking of why does the Ecto layer matter (aside from DB):

1. Error format compatibility. Absinthe, Phoenix forms, and LiveView all expect %Ecto.Changeset{errors: [...]} -- that's the ecosystem's error rendering pipeline. As awesome as Zoi's error format is, using it directly means every app needs adapters/converters at each boundary. The Zoi changeset output would plug into this pipeline without any such hand-rolled adapters.
2. Gradual adoption. Any existing Phoenix app can adopt Zoi one resolver / consumer / controller at a time. Each place you switch to Zoi.Ecto still produces a Changeset that the rest of the codebase understands i.e. no big-bang migration needed.
3. Compile-time struct guarantees. Struct field access, pattern matching, compile-time key checks -- a plain map from Zoi.parse works but loses all of these. I want to go all the way and have as much compile-time checks as possible.

So Zoi "owns" validation and coercion, but converts to Ecto.Changeset that the broader Phoenix / Absinthe ecosystem already knows how to work with, is I suppose an even more summarized version.

btw I'm thinking of putting together a small demo project where I can show you valid use-cases and exercise the PR against realistic scenarios. Still not sure I can spare that much extra time but it might be helpful to you. WDYT?

[phcurado]

Understood, basically projects using embedded_schema for boundary validation could benefit from Zoi if this lib offers some bridge between embeds. On my side I just use zoi directly for that but understand that the current ecosystem and projects are already locked with changesets.

Btw creating the changeset wrapper on top of Zoi was not that difficult, logic was in a few lines: https://github.com/phcurado/zoi/pull/159/changes#diff-f774c8d2e94426c15e770fc52f42902fce2980577459f83d365870bac1ebd59cR204
Hope this can remove some of the barriers between both libs.

I would say all points are valid besides the third one. We can have compile time guarantees with zoi as well using module attributes when declaring the schema. Every schema can raise if you don't declare with correct options. Also Zoi supports structs so it would be similar (in a sense) to embeds.

So in a sense, you could replace embedded_schema with Zoi, if zoi also offers way to convert to changeset.

defmodule MyApp.Onboarding do
  @schema Zoi.discriminated_union(:product_type, [
            Zoi.map(
              %{
                product_type: Zoi.literal(:card_type_1),
                partner_account_number: Zoi.string(),
                initial_balance: Zoi.integer() |> Zoi.min(0),
                origination_date: Zoi.date(coerce: true),
                promo_end_date: Zoi.date(coerce: true) |> Zoi.optional()
              },
              empty_values: [nil, ""]
            ),
            Zoi.map(
              %{
                product_type: Zoi.literal(:card_type_2),
                last_four: Zoi.string() |> Zoi.min(4) |> Zoi.max(4)
              },
              empty_values: [nil, ""]
            )
          ])

  def parse(params) do
    case Zoi.parse(@schema, params) do
      {:ok, parsed} -> {:ok, parsed}
      {:error, errors} -> {:error, Zoi.Ecto.errors_to_changeset(errors)}
    end
  end
end

This implements all the business rules you sent it (the sample code) and everything is 100% on the schema. And any additional validation that is more complex we could use refinement api. So you would not have to deal with embeds at all unless strictly necessary. Replacing embeds with zoi schema should be quite easy (specially with AI). Also I could support discriminated_union in the Zoi.struct like Zoi.discriminated_struct or somehting similar if you want the struct via compile time.

As for the project with more realistic scenarios, all good for me I can take a look. If you don't have time you can also add directly in your codebase and try little by little with the existent business rules there and see if this library is helpful or we need to change more, and I will be happy to do it.

PS: if we move forward with embeds I might put the logic in another library (like zoi_ecto) because it's a bit more specific and tied to ecto, I'm doing here to validate things. If we go with something simpler like discriminated_struct and a simple to_changeset function it can still be here on this lib since it's a simpler wrapper.

[dimitarvp]

Yep, I over-focused on embedded Ecto.Schemas -- thanks for finding the better MVP. I agree that just translating errors to an Ecto.Changeset would be enough for my current use-case. Though I am yet to go to your PR and download it and try it -- that's on me to experiment with, soon.

So we can keep your current work focused only on that and then have a follow-up PR for generating embedded schemas later? I am asking the latter because I've also seen quite a lot of projects where the embedded schemas and the changesets from their validation are passed around to everything: dead-view / LiveView forms, cast_embed, to Phoenix templates. But that can be parked for later, yep.

And I agree such an effort should be an adjacent library simply because you really don't want to tie your core library to Ecto -- and because once you start integrating, the bridge might grow too much for comfort if it was in the core library. So yeah, I'd do the same.

If you don't have time you can also add directly in your codebase and try little by little with the existent business rules there and see if this library is helpful or we need to change more, and I will be happy to do it.

Sadly I can barely scrape the time to interact with you currently and if you are interested in following up with this I'd be very grateful.

[phcurado]

No worries the discussion already gave me plenty of input. We can always come back to the topic late, happy to help.
I can support changesets here on this lib, and embedded schemas most probably will be an adjacent library as mention. I can create the library once we sort things out on the features you need. I don't have any application on my side with this setup so it's hard for me to validate.

Sadly I can barely scrape the time to interact with you currently and if you are interested in following up with this I'd be very grateful.

yes I'm interested in following up with this

[dimitarvp]

I have two current uses for Zoi:

1. Normal use-cases where I need zero extensions to the library;
2. Use-cases where I need Zoi to produce an Ecto.Changeset with errors translated from Zoi. As mentioned before, I am OK with the fact that this is a lossy compression of sorts.

Let's agree on the following: allow me little time to play with your PR and give you exact feedback on how well does it serve my item 2 above -- let's together iterate on this until we are both satisfied. The rest can wait (generating embedded schemas and/or having a playground mini app).

Would you agree with this kind-of-a-plan?

[phcurado]

I agree 🚀

[dimitarvp]

I would like to contribute only errors_to_changeset/1 but that still deals with Ecto and you might not want that inside the core Zoi. I will open a PR Soon™ but -- would you agree to add this to core Zoi or you'll use that PR as a base to start a mini zoi_ecto library? Just curious. I'll offer the PR to core Zoi regardless.

[phcurado]

we can have it on this library, using ecto as an optional dependency like I did in my PR.
Module would be Zoi.Ecto and would have changeset helpers (such and maybe only errors_to_changeset/1).
As for separating into another library, this is a thin line if we should or not so my perspective is:

• Zoi have integration with phoenix forms already as optional dependency
• Zoi have integration with Decimal library as optional dependency
• Zoi is more useful having more integrations with elixir ecosystem

Lastly, and the main point I would say, is that it's such a small function that integrates with one of the most popular libraies would be fine to have it as long as we put the dependency optional. I can separate in future and I would do it if the implementation starts to outgrown this library (like embedded schemas, schemas itself,etc is too specific to ecto while changesets are more generic). In that case I would add a deprecation notice for anyone using the Zoi.Ecto module, so they use the new zoi_ecto lib if we go to this path.

Anyways answering the question I'm open to a PR. It's also possible to make it more generic but not sure if you need this, like:

schema = Zoi.map(%{first_name: Zoi.string()})
Zoi.Ecto.changeset(schema, %{first_name: "Paulo"})
# Parses with Zoi and wraps the result into a changeset, returning errors or the parsed data inside the changeset struct

Most probably this Zoi.Ecto.changeset(schema, %{first_name: "Paulo"}) would still use the function under the hood you are proposing errors_to_changeset so it's fine to open a PR with only this function.

[dimitarvp]

Opened a PR: #162. Let's discuss there.