[R] processing nanoarrow streams with geoarrow column

[JosiahParry]

I have the ability to return a nanoarrow stream with a geoarrow geometry array. I'd like to be able to take the stream and turn it into a tabular data structure (data.frame-esque of any variety). Is it possible with geoarrow as it is today, to take this and turn it into a data.frame with a geoarrow array column?

Below is a small reprex using an R package im trying to develop using arrow-rs and geoarrow-rs
https://github.com/JosiahParry/serde_esri

library(serdesri)

url <- "https://services.arcgis.com/P3ePLMYs2RVChkJx/arcgis/rest/services/ACS_Population_by_Race_and_Hispanic_Origin_Boundaries/FeatureServer/2/query?where=1=1&outFields=objectid&resultRecordCount=10&f=json"

req <- httr2::request(url)
resp <- httr2::req_perform(req)
json <- httr2::resp_body_string(resp)

stream <- parse_esri_json_str(json, 2)
stream
#> <nanoarrow_array_stream struct<OBJECTID: int64, geometry: geoarrow.polygon{large_list<rings: large_list<vertices: fixed_size_list(2)<xy: double>>>}>>
#>  $ get_schema:function ()  
#>  $ get_next  :function (schema = x$get_schema(), validate = TRUE)  
#>  $ release   :function ()

df <- as.data.frame(stream)
#> Warning in warn_unregistered_extension_type(x): geometry: Converting unknown
#> extension geoarrow.polygon{large_list<rings: large_list<vertices:
#> fixed_size_list(2)<xy: double>>>} as storage type
#> Warning in warn_unregistered_extension_type(storage): geometry: Converting
#> unknown extension geoarrow.polygon{large_list<rings: large_list<vertices:
#> fixed_size_list(2)<xy: double>>>} as storage type

str(df, 1)
#> 'data.frame':    10 obs. of  2 variables:
#>  $ OBJECTID: num  1 2 3 4 5 6 7 8 9 10
#>  $ geometry: list<list<list<dbl>>> [1:10]

[paleolimbot]

I'm not quite there yet but the PR that implements most of it is here and I'll try to smooth out the rough edges in the next few days #75 .

Basically, it will let a list(nanoarrow_array) masquerade as a data.frame column/vctr. It will survive slicing (e.g., head(1:X) but not rearranging (e.g., arbitrary filter/take) quite yet, but that will let you do as.data.frame() and then choose the destination you want (e.g., sf::st_as_sfc(), or whatever).

Also, geoarrow for R doesn't quite do 64-bit offsets (i.e., the large_list bit of your output above). If there's an option to use 32-bit offsets in geoarrow-rs that is pretty much required right now for anything geoarrow-c based. You'll also get better performance if you use separated/struct coordinates in R (because it more closely matches wk::xy() and sf::st_sfc()'s memory layout).

[JosiahParry]

amazing!