Add a grib/grib2 reader

[jrleeman]

A highly requested feature to be able to natively read grib and grib2.

[darothen]

I'd +1 this; currently, you can ingest grib files in xarray via PyNIO. If PyNIO can be updated to Python 3 (see NCAR/pynio#10) then I think all the work is done and everyone (MetPy and beyond) can lean on that.

[dopplershift]

I want to do a pure Python version regardless, but that's because I'm a broken individual--I like solving the puzzle that is decoding a binary file format.

[darothen]

@dopplershift you're a brave soul :) I did that for darothen/xbpch and I'll be very happy to not touch any similar job for quite some time.

[dopplershift]

We'll see how I feel when I get to it...I'm about 50%-60% done on BUFR.

[nawendt]

This is something I have played around with a little bit and think I could help get finished. I'd be interested in seeing what sort of structure you had in mind for a reader like this.

[dopplershift]

@nawendt (sorry this response got lost in my inbox) I think the structure I'm looking for is something that integrates well with Xarray. In general this means:

1. Scanning through the big grib file to see what message are available (forming an index)
2. Setting up Xarray with the appropriate variables
3. Getting the binary data from one or more messages when the variable is indexed
4. Translating GRIB time/coordinate information into proper xarry dimensions and coordinates

[nawendt]

@dopplershift Thanks. I've also been thinking about how to implement GRIB tables. Dictionaries would be straight-forward, but with some tables referencing other tables for given parameters, that may not work well. Could a database type structure work where tables are linked? I am not too familiar with what is available in that regard. Ultimately, users will have to be able to add their own custom tables no matter what the solution is.

[dopplershift]

I think probably best to have an explicit function/method call to point to a custom table, rather than any kind of "autoload". It's supporting formatting for those different tables that's going to be the real challenge.

My inclination is to be able to read directly the GRIB tables from the WMO so that we can upgrade easily. If this is too slow for too many tables, we could look at storing in an sqlite database, since that's built into Python. Would probably be best to have MetPy build that locally and cache on a user's machine.

As far as the actual data structures for tables are concerned...maybe sqlite would actually be best. It's probably going to take some experimentation there. Would be a useful first start just to get the table management and access API squared away.

[eliteuser26]

Accessing the Pynio module will not be a good solution as there is no Windows version made of the module. Also according to the Pynio group they won't develop a Windows version which is unfair. The better solution would be to decode the binary data for the Grib/Grib2 files. In this case the data can be converted into an XArray to be used in Metpy if it is possible. I have done the research on this. I am able to decode a Grib2 binary data in Python where I can visualize all the different codes in each table. I started to convert the different tables into dictionaries into a single python file where I can load them as necessary.

[blaylockbk]

Check out this project: https://github.com/ecmwf/cfgrib

[zbruick]

Is this still on the list to add to MetPy, now that Xarray can handle GRIB? Tangentially, this should probably be an example in the Gallery.

[dopplershift]

I would still love to have a pure Python reader, but this is certainly less of a priority given that cfgrib exists.

[eliteuser26]

There are still limitations as seen from the documentation. Still no Conda or pip version. It is indicating as work in progress. No Windows version only Linux or MacOS version with Linux. I did find a way to decode the Grib data with Python only but having problems reading Jpeg2000 data. Only supported under OpenJPEG library for Linux.