Using Planetary Computer to scale analysis of NAIP scenes

[carmengg]

Hi!

I’d like to ask about recommendations to efficiently use the Planetary Computer to scale up our current project. We have implemented a workflow on the PC that uses a random forest model to do a per-pixel classification on a given NAIP scene, looking for a particular invasive plant species. Here's the code.

We want to run our classification model on all the NAIP scenes covering the Santa Barbara County coast in a given year. There are about 25 scenes per year, and the issue is they overlap. In terms of leveraging the PC’s strengths, would it be better to :

1. first create a big raster mosaic of the NAIP scenes (maybe using gdal_merge.py?) and then run the model over this very big raster, or
2. run each NAIP scene through the model and then make a mosaic of the model’s results?

For option (1), we also need to keep track of the NAIP collection data for the scene before feeding the image into the model, so each pixel would be associated to a date in the mosaic. For option (2), the rasters resulting from the model have a single integer band.

Any suggestions on how to do this most efficiently would be appreciated. It could also be there’s an option (3) we haven’t considered. I'd really appreciate any insight you might have about this!

[TomAugspurger]

A few quick thoughts, I'll hopefully be able to take a deeper look later:

1. I'm not sure how much NAIP scenes overlap, but it'd probably be slightly more efficient to mosaic and then predict. That said, the predictions will be smaller (1 band instead of 4) so I'm not 100% confident in that since I don't know how the mosaicing scales with the number of bands. It'll probably be easier to predict and then mosaic though, since you won't need to store an intermediate mosaic, so maybe that'd be best.
2. It might be worth thinking about whether or not you need a single mosaic output. Or would it be acceptable to just have one output COG per input scene? Then you don't need to worry about mosaicing at all, and everything can be completely parallelized at the scene level.
3. Glancing at your code, it looks like you load the data on the client and then (possibly) do the prediction on the cluster. This will be slower than loading the data with Dask since you're client will be a bottleneck.

[mjevans26]

Appreciating the insight on this thread. I'm curious what the current best practice is for stacking/mosaicking NAIP data accessed through the planetary computer? Following the original documentation, we've been using a workflow with stac_vrt and rioxarray. Recently this has been resulting in unexpected behavior - specifically 409 errors when values are read out of the datarray. For completeness, we are creating 1200x1200 pixel NAIP chips for ingestion into a CNN and running this inferencing over counties, so I think the input data needs to be mosaicked. Any guidance is much appreciated!

[777arc]

Apologies for the delay. Regarding the "Recently this has been resulting in unexpected behavior", was it working at one point, and is the only thing that changed the data from planetary computer? Thanks!