r/sdplyr: Can `collect()` return an `sf` object by default?

[yutannihilation]

I know this would be probably more complex than it sounds.

• The result data frame might not contain a geometry column. If we naively attempt to create an sf object, it fails when there's no geometry column. We need to fall back to tibble.
• The result data frame might contain multiple geometry columns. We might need to provide some option to specify which column is primary.
• Considering SedonaDB has both vector and raster, honestly I'm not sure sf is the appropriate representation.

[paleolimbot]

It's a good question. I didn't think too hard when I was first writing the bindings and there isn't a ton of thought put in to the current default (a tibble with a GeoArrow shell column, that possibly should be ALTREP instead of the S3 madness that it is). I'm hesitant to materialize to sf by default because that materialization is slow and creates a lot of SEXPs, which can slow down the garbage collector considerably when we get millions of rows. Sometimes the next thing is Arrow anyway, where the current output would serve zero copy.

We could add another function sd_collect_sf(), an option to collect(), or something else if that behaviour is useful!

[yutannihilation]

materialization is slow and creates a lot of SEXPs

Yeah, but I haven't succeed to find a bare data.frame with a GeoArrow column useful yet... For example, we cannot plot it with ggplot2, or manipulate it with sf. Also, once we collected it as a data frame, we cannot convert it to sf by sf::st_as_sf(), so I end up re-running the whole SedonaDB SQL just to call sf::st_as_sf().

This might be simply because I'm not familiar with GeoArrow ecosystem, but when is it actually useful?

[yutannihilation]

when is it actually useful?

Never mind. I know it's useful for zero copy interop. Thinking this again, I think my question is about the semantics of collect() rather than the usefulness of GeoArrow itself. I thought it's something like "materialize at all cost". Loading all the data into R's side so that we can manipulate it on R's side.

[paleolimbot]

Also, once we collected it as a data frame, we cannot convert it to sf by sf::st_as_sf()

This is something has always bothered me about sf...there are a lot of good (and considerably faster) ways to materialize a geometry column (wk, geos, sf, s2, geoarrow) and st_as_sf.data.frame() recognizes none of them. Of course, the tooling around the others (e.g., for ggplot2) isn't there.

My two other gripes with sf is that there's no way to represent geography and the geometry semantics with lon/lat are different (unless sf_use_s2 is false)...also the sticky geometry column makes transformations unpredictable if you're writing generic code against its result.

How about a geometry_ptype and geography_ptype argument for collect that can be set with options()? If the option is popular enough we can change the default.