How to download downscaled CMIP6 data

[heroldn]

Hi, apologies if this is a beginner question but after 20 minutes of looking I can't find a straight forward way to simply download files I'm interested in. I don't need to do any calculation, I simply want a local copy. Is there an example of how to do this, or a server that I can wget from? I see storage.ipynb on how to write to "blob storage", which seems potentially close, but nothing on how to browse what's available on said storage. Thanks again, these are amazing products (the data and the system to interact with it).

[TomAugspurger]

We do tend to encourage loading files directly into memory in your application (whatever that is for your use-case).

But once you have a signed URL you can access the file like any other. For example, from Python:

import urllib.request

filename, _ = urllib.request.urlretrieve(<signed_href>)

Or with curl:

curl -O '<signed_href>'

or you can use something like azcopy.

IIRC, we host a couple CMIP6 downscaled datasets in a couple different formats. nasa-nex-gddp-cmip6 is a bunch of NetCDF files. The various CIL-GDPCIR datasets are stored as Zarr. For Zarr, you'd probably want to use azcopy since a single Zarr group is made up of a bunch of files.

but nothing on how to browse what's available on said storage.

Storage Explorer or your Azure Portal are probably the best way to browse your storage containers.

[mfwehner]

I have essentially the same question, but I don't understand the answer. I created a netcdf file and tried "copy download link" and "copy shareable link", but when I wget that copied string on my supercomputer, it returns a text file not the nc file. So I tried os.system('scp myfile.nc myusername@dtn01.nersc.gov:mypath') within a PC python window but it returned "Host key verification failed." Any suggestions?

[TomAugspurger]

I created a netcdf file and tried "copy download link" and "copy shareable link",

@mfwehner can you confirm that this is a NetCDF file you created on your home directory in Hub and now you're trying to download it locally (outside the hub)? There might be something in the JupyterHub documentation about how to accomplish that, but I probably wouldn't that as a general approach. Instead, I'd recommend writing your file to a storage system that can be accessed remotely, like Azure Blob Storage.

[mfwehner]

On Nov 28, 2022, at 1:10 PM, Tom Augspurger ***@***.***> wrote: I created a netcdf file and tried "copy download link" and "copy shareable link", @mfwehner <https://github.com/mfwehner> can you confirm that this is a NetCDF file you created on your home directory in Hub <https://planetarycomputer.microsoft.com/docs/overview/environment/> and now you're trying to download it locally (outside the hub)?

Yes. Same problem when I tried a text file

There might be something in the JupyterHub documentation about how to accomplish that, but I probably wouldn't that as a general approach. Instead, I'd recommend writing your file to a storage system that can be accessed remotely, like Azure Blob Storage.

I looked at some of the Azure Blob Storage but not enough to figure out how to write a netcdf file to it. But the download command to transfer the file to our mainframe cori.nersc.gov, would appear from https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-files to be azcopy copy 'https://mystorageaccount.file.core.windows.net/myfileshare/myTextFile.txt?sv=2018-03-28&ss=bjqt&srs=sco&sp=rjklhjup&se=2019-05-10T04:37:48Z&st=2019-05-09T20:37:48Z&spr=https&sig=/SOVEFfsKDqRry4bk3qz1vAQFwY5DDzp2%2B/3Eykf/JLs%3D' 'C:\myDirectory\myTextFile.txt' --preserve-smb-permissions=true --preserve-smb-info=true Installing azcopy on a Cray XC40 is not a task that I could do, or would be permitted to do for that matter. Are there standard UNIX commands to access Azure Blob Storage? Thanks Michael

…

— Reply to this email directly, view it on GitHub <#98 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACSKCHDK3QI7VDLHZOXXQUTWKUNUHANCNFSM574VSDQA>. You are receiving this because you were mentioned.

[TomAugspurger]

The easiest way to upload the blob from Python is to write the file to a local disk and use azure.storage.blob to upload it: https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-python?tabs=managed-identity%2Croles-azure-portal%2Csign-in-azure-cli#upload-blobs-to-a-container.

Blobs in Azure Blob Storage can be accessed over HTTP (with a SAS token in the URL for authentication), so you can just use curl / wget to access the files.

[mfwehner]

OK, Thanks. I may be making this harder than it needs to be. All I really want to do is to download a few variables from the NEX-GDDP-CMIP6 database. Which is already on the azure blob storage system, I think. Michael

…

On Nov 28, 2022, at 1:33 PM, Tom Augspurger ***@***.***> wrote: The easiest way to upload the blob from Python is to write the file to a local disk and use azure.storage.blob to upload it: https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-python?tabs=managed-identity%2Croles-azure-portal%2Csign-in-azure-cli#upload-blobs-to-a-container <https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-python?tabs=managed-identity%2Croles-azure-portal%2Csign-in-azure-cli#upload-blobs-to-a-container>. Blobs in Azure Blob Storage can be accessed over HTTP (with a SAS token in the URL for authentication), so you can just use curl / wget to access the files. — Reply to this email directly, view it on GitHub <#98 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACSKCHDJEROGUPLIOZ2INMDWKUQKTANCNFSM574VSDQA>. You are receiving this because you were mentioned.