Clearer errors in case of incorrect authentication

[veenstrajelmer]

Is your feature request related to a problem? Please describe.

When using an old url or an incorrect apikey, the errors/warnings that cdsapi gives are not clear. This happens to several users, for instance in https://forum.ecmwf.int/t/cdsapi-key-and-404-issues/10905.

Describe the solution you'd like

To avoid users getting confused, I generated authentication functions for cdsapi in my own toolbox (including tests). It would be great if these could be (at least partially) transferred to cdsapi itself. The code is in here:
https://github.com/Deltares/dfm_tools/blob/af2ea58f8ebfc937274c92dc1a42854d50d6dc22/dfm_tools/download.py#L105-L195

The corresponding tests are here:
https://github.com/Deltares/dfm_tools/blob/af2ea58f8ebfc937274c92dc1a42854d50d6dc22/tests/test_download.py#L26-L213

Describe alternatives you've considered

Any alternative method that generates clear errors would suit. For instance, this function sounds promising:

import cdsapi
c = cdsapi.Client()
c.check_authentication()

However, it raises "AttributeError: 'LegacyApiClient' object has no attribute 'check_authentication'" at least half of the time.

Additional context

Authentication check with import cdsapi; c = cdsapi.Client(); c.client.check_authentication() often fails inconveniently, useful errors would be helpful. These usecases follow the same order as the test_cds_credentials_* tests in test_download.py of dfm_tools:

• in case of authentication via CDSAPI_KEY envvar instead of rcfile: Exception: Missing/incomplete configuration file: C:\Users\veenstra/.cdsapirc >> would be useful if default url would be assumed instead. However, this case does succeed if the envvar CDSAPI_URL is also supplied.
• in case of new url and incorrect apikey (via rcfile or envvars): HTTPError: 401 Client Error: Unauthorized for url: https://cds.climate.copernicus.eu/api/profiles/v1/account/verification/pat Authentication failed >> acceptable error/message
• in case of old v2 url and incorrect apikey (via rcfile or env vars): HTTPError: 404 Client Error: Not Found for url: https://cds.climate.copernicus.eu/api/v2/profiles/v1/account/verification/pat >> acceptable error message
• in case of old cds-beta url and incorrect apikey (via rcfile or envvars): SSLError: HTTPSConnectionPool(host='cds-beta.climate.copernicus.eu', port=443): Max retries exceeded with url: /api/profiles/v1/account/verification/pat (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1006)'))) >> this is not super useful, discontinue the url or fix the certificate
• in case of new url and apikey containing : (this was normal with old CDS) (via rcfile or envvars): AttributeError: 'Client' object has no attribute 'client' >> this is not helpful, why is the client not correctly set up, would be great if check_authentication() would be available but raise an error in this case

In short:

• the failing certificate (SSLError) makes error handling more cumbersome. Only catching HTTPError (401/404/etc) is easier
• with old apikeys containing : the check_authentication() function cannot be reached, this is inconvenient.
• optional: in case of envvars, it would be useful if the default cds url is assumed if no CDSAPI_URL envvar is supplied

Organisation

Deltares

[veenstrajelmer]

@EddyCMWF