Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

errors='ignore' for parse_cf? #1917

Open
andhuang-CLGX opened this issue Jun 17, 2021 · 3 comments
Open

errors='ignore' for parse_cf? #1917

andhuang-CLGX opened this issue Jun 17, 2021 · 3 comments
Labels
Area: Units Pertains to unit information Area: Xarray Pertains to xarray integration Type: Bug Something is not working like it should

Comments

@andhuang-CLGX
Copy link

import os
from io import BytesIO

import s3fs
import metpy
import xarray as xr

fs = s3fs.S3FileSystem(anon=True)
fs.ls('s3://noaa-goes16/')

files = fs.ls('noaa-goes16/ABI-L1b-RadC/2019/240/00/')
with fs.open(files[0], 'rb') as f:
    ds = xr.open_dataset(BytesIO(f.read()), engine='h5netcdf')

ds.metpy.parse_cf()

If I do this, it crashes, but if I do this, it works:

import os
from io import BytesIO

import s3fs
import metpy
import xarray as xr

fs = s3fs.S3FileSystem(anon=True)
fs.ls('s3://noaa-goes16/')

files = fs.ls('noaa-goes16/ABI-L1b-RadC/2019/240/00/')
with fs.open(files[0], 'rb') as f:
    ds = xr.open_dataset(BytesIO(f.read()), engine='h5netcdf')

ds.drop(["x_image", "y_image"]).metpy.parse_cf()
@andhuang-CLGX andhuang-CLGX added the Type: Bug Something is not working like it should label Jun 17, 2021
@andhuang-CLGX
Copy link
Author

Actually, still doesn't work:
ds["Rad"].metpy.cartopy_crs
Proj4Error: Error from proj: b'major axis or radius = 0 or not given'

@andhuang-CLGX
Copy link
Author

I was able to recreate the issue. metpy may need to call .item() if the len(attrs["something"]) == 1

proj_info = ds['goes_imager_projection'].attrs
globe = ccrs.Globe(
    semimajor_axis=proj_info["semi_major_axis"].item(),
    semiminor_axis=proj_info["semi_minor_axis"].item()
)
geo_crs = ccrs.Geostationary(
    central_longitude=proj_info["longitude_of_projection_origin"].item(),
    satellite_height=proj_info["perspective_point_height"].item(),
    globe=globe,
)

@jthielen
Copy link
Collaborator

So it looks there are a few different issues going on here!

1) parse_cf() should be more error safe

When I used this file downloaded from S3 rather than BytesIO reading, this example seems to fail with TypeError: unsupported operand type(s) for -: 'ParserHelper' and 'int', something very unhelpful for the end user. This (along with a recent stackoverflow question) makes me think that more robust error handling inside parse_cf, particularly with regards to units, is in order.

2) A UDUNITS-valid expression failed our tweaked pint registry

In this dataset, the variable kappa0 has has a units attribute of (W m-2 um-1)-1 which I believe to be valid UDUNITS format, but something that our regex below failed to capture:

MetPy/src/metpy/units.py

Lines 35 to 39 in c7b0b9b

functools.partial(
re.sub,
r'(?<=[A-Za-z])(?![A-Za-z])(?<![0-9\-][eE])(?<![0-9\-])(?=[0-9\-])',
'**'
),

While that regex borders on "write-only code", I'll take a wild guess and hope the following modification may do the trick?

r'(?<=[A-Za-z\)])(?![A-Za-z\)])(?<![0-9\-][eE])(?<![0-9\-])(?=[0-9\-])'

(addition of \) to the look-behinds)

xref #1362

3) The Proj4Error

I wasn't able to replicate this using the local netcdf file and default netcdf engine, so I think this may be something weird with the h5netcdf engine. Due to time constraints, I'll let someone else chime in on that one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: Units Pertains to unit information Area: Xarray Pertains to xarray integration Type: Bug Something is not working like it should
Projects
None yet
Development

No branches or pull requests

3 participants