Daily Temporal Climatology using xcdat #469
Replies: 4 comments 15 replies
-
|
I would recommend checking if you can get the results you need with smaller datasets, or rewriting your script and algorithm to loop on smaller subsets of the data On Linux, you could also try to use cdo to reduce your data before loading it in your python script. They seem to provide some Win binaries that you could try to use |
Beta Was this translation helpful? Give feedback.
-
Thanks for this, but I have clipped the global data to just a small box with 20 latitudes. However, chunking the data as mentioned here helped and produced the temporal climatology. |
Beta Was this translation helpful? Give feedback.
-
|
I have this problem a lot.
If it was me, I would divide up the calculation spatially. If latitude-longitude data, I would load one latitude line in at a time via a loop.
Michael
… On May 10, 2023, at 2:19 AM, Jean-Yves Peterschmitt ***@***.***> wrote:
117. GiB is indeed quite big, and would be a problem even on a Linux cluster!
I would recommend checking if you can get the results you need with smaller datasets, or rewriting your script and algorithm to loop on smaller subsets of the data
On Linux, you could also try to use cdo <https://code.mpimet.mpg.de/projects/cdo> to reduce your data before loading it in your python script. They seem to provide some Win binaries <https://code.mpimet.mpg.de/projects/cdo/files> that you could try to use
—
Reply to this email directly, view it on GitHub <#469 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACSKCHGX57VKIXNA75GH7V3XFNMTHANCNFSM6AAAAAAXYVWWGA>.
You are receiving this because you are subscribed to this thread.
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks alot. Defaulting to CF 'standard' calendar. So, does it mean it has by default used the standard calendar now? My ds.time.attrs display: (this is for the climatology generated from the temporal climatology function of xcdat) |
Beta Was this translation helpful? Give feedback.
-
|
NetCDF writes time in units like "days since 2000-01-01" (the unit, days, and reference time, 2000-01-01, can change). The interpretation of time depends on the calendar embedded in the dataset (e.g., proleptic_gregorian, 360_day, etc.). For this reason, it is important to know the calendar. Your dataset must not include the calendar. This warning message basically says that xcdat is assuming that you are using a time = ds.time
attrs = time.attrs
attrs['calendar'] = 'standard'
ds['time'] = time |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, my original data looks like: (attaching screehshot just because it can give a glimpse of format) Also in the first screenshot, although the warning appeared, xcdat_boundas are True(ref: screenshot 1). After assigning, I cannot see that xcdat_bounds thing in my data variables list. Second thing: dsc_seas = dsc.temporal.climatology(
"icec",
freq="season",
weighted=True,
season_config={"dec_mode": "DJF", "drop_incomplete_djf": True},
)and my dsc_seas.icec.time array looks like: Ref: Calendar is standard here. What could be the error? |
Beta Was this translation helpful? Give feedback.
-
|
Okay, it could be becuase the code seems to divide something by zero/nan. |
Beta Was this translation helpful? Give feedback.
-
|
@GIRIJA-KALYANI - I don't fully understand the first probelm – you were setting the calendar to get rid of the warning message. Presumably the calendar you set shows up in This thread is becoming a little difficult to follow. I would really appreciate if you shared formatted, reproducible code snippets and briefly summarize what the problem is and what you tried to do to resolve the problem. If you searched for similar errors and tried things from stackoverflow or looked at matplotlib/xarray documentation, you can link to those (and share code that did not help). |
Beta Was this translation helpful? Give feedback.
-
|
I'm trying to replicate the seasonal climatologies as https://github.com/xCDAT/xcdat/blob/main/docs/examples/climatology-and-departures.ipynb For seasonal climatology: I tried the follwoing dsc_seas = dsc.temporal.climatology(
"icec",
freq="season",
weighted=True,
season_config={"dec_mode": "DJF", "drop_incomplete_djf": True},
)which returns the follwoing error: My seasonal climatology data (given below) although returned 4 time variables (4 seasons), while plotting displays the Runtime Warning with blank plots My Goal: Seasonal nd Daily anomalies using the xcdat temporal climatology. |
Beta Was this translation helpful? Give feedback.
-
|
@GIRIJA-KALYANI - could you try to figure out if this is a plotting issue or a data issue? I appreciate the additional context, but it would be extremely helpful if you could provide a complete minimal example leading to the issue (see the example below or a minimal complete verifiable example. From your notes above, I can't really tell if there was a problem in calculating the seasonal climatology or a problem in plotting something (you don't share your plot command or what example filewget https://downloads.psl.noaa.gov/Datasets/noaa.oisst.v2.highres/icec.day.mean.1982.ncpython example# imports
import xcdat as xc
import matplotlib.pyplot as plt
# open dataset
ds = xc.open_dataset('icec.day.mean.1982.nc')
# climatology calculation
dsc = ds.temporal.climatology("icec", freq="season", weighted=True, season_config={"dec_mode": "DJF", "drop_incomplete_djf": True})
# plot
plt.pcolor(dsc.icec[0])
plt.colorbar()
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks for this, I'm using the same data. For 40 years!!! ice_daily = xc.open_mfdataset('/icec/*.nc', combine = 'nested', concat_dim="time") # reads all the files in the folderAlthough gives the following error: Does the job perfectly as we can see time:15097 Now, clipped to my ROI: rs = ds.sel(lon = slice(160, 200),lat =slice(-80, -73))Looks like: Temporal Climatology day wise: dsclim = dsc.temporal.climatology("icec", freq="day") #worked finebut when I want to display: The same for seasonal climatology as well: dsc_seas = dsc.temporal.climatology(
"icec",
freq="season",
weighted=True,
season_config={"dec_mode": "DJF", "drop_incomplete_djf": True},
)but cant visualise it. All I want is: solve the plotting issue. I tried putting up the issue the best way I can. This issue was not before though. |
Beta Was this translation helpful? Give feedback.
-
|
Well, after restarting the system, although the error persists, I can view the maps. |
Beta Was this translation helpful? Give feedback.
-
|
That is great news – thank you for following up. I was confused, because these are warnings (not errors) so it seems like the calculation is (or should be) completing. I think there may be lingering memory issues since this dataset is so large – this may be why the restart helped. I think we're going to explore xcdat + dask in the future, so we may add more documentation about this. We'll keep in mind large datasets as something to contend with. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for the discussion. I added some minor edits to some of text from @GIRIJA-KALYANI, formatting code and log messages for easier read of this thread for the future reference. |
Beta Was this translation helpful? Give feedback.
-
|
That would be great, seriously.
Climate data generally deals with such huge chunks, but xcdat provides
easier and accurate solutions.
Exploring other API too from the module.
Thanks alot for the response again.
*Girija Kalyani BuradaDoctoral StudentSchool of Geography, Environment and
Earth SciencesVictoria University of WellingtonNew Zealand*
…On Thu, May 11, 2023 at 6:45 PM Stephen Po-Chedley ***@***.***> wrote:
That is great news – thank you for following up. I was confused, because
these are *warnings* (not *errors*) so it seems like the calculation is
(or should be) completing. I think there may be lingering memory issues
since this dataset is so large – this may be why the restart helped. I
think we're going to explore xcdat + dask in the future, so we may add more
documentation about this. We'll keep in mind large datasets as something to
contend with.
—
Reply to this email directly, view it on GitHub
<#469 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABWDHEHYS2R5OIR6BJSNCATXFSDJRANCNFSM6AAAAAAXYVWWGA>
.
You are receiving this because you modified the open/close state.Message
ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Awesome, glad to hear you got things working and |
Beta Was this translation helpful? Give feedback.
-
Hi there,
I have daily data for about 40 years, each year have been imported using mf_datset and nested along 'time' dimension.
Trying to use Daily temporal climatology on this, and it shows memory issues.
The error reads as follows:
Can you please provide insights on how to handle it?
Many Thanks in advance.
Also, how can I apply dask chunks on this.
My data looks like this:
Beta Was this translation helpful? Give feedback.
All reactions