Skip to content

Commit

Permalink
Refactor settings
Browse files Browse the repository at this point in the history
  • Loading branch information
@sandorkertesz
sandorkertesz committed Jan 16, 2025
1 parent 8c7c1a6 commit 4e2c14d
Show file tree
Hide file tree
Showing 42 changed files with 921 additions and 533 deletions.
70 changes: 41 additions & 29 deletions docs/examples/cache.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
"metadata": {},
"outputs": [],
"source": [
"from earthkit.data import settings, cache"
"from earthkit.data import config, cache"
]
},
{
Expand All @@ -39,7 +39,7 @@
"source": [
"earthkit-data uses a dedicated **directory** to store the results of remote data access and some GRIB/BUFR indexing information. By default this directory is **unmanaged** (its size is not checked/limited) and **no caching** is provided for the files in it, i.e. repeated calls to :func:`from_source` for remote services and URLSs will download the data again!\n",
"\n",
"When :ref:`caching <caching>` is **enabled** this directory will also serve as a :ref:`cache <caching>`. It means if we run :func:`from_source` again with the same arguments it will load the data from the cache instead of downloading it again. Additionally, caching offers **monitoring and disk space management**. When the cache is full, cached data is deleted according to the settings (i.e. oldest data is deleted first). "
"When :ref:`caching <caching>` is **enabled** this directory will also serve as a :ref:`cache <caching>`. It means if we run :func:`from_source` again with the same arguments it will load the data from the cache instead of downloading it again. Additionally, caching offers **monitoring and disk space management**. When the cache is full, cached data is deleted according to the configuration (i.e. oldest data is deleted first). "
]
},
{
Expand All @@ -53,18 +53,24 @@
"tags": []
},
"source": [
"In the examples below we will change the settings multiple times. First we ensure all the changes are temporary and no settings are saved into the configuration file. We also reset the settings to the defaults."
"In the examples below we will change the configuration multiple times. First we ensure all the changes are temporary and no options are saved into the configuration file. We also reset the configuration to the defaults."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "71214b97-2b64-442d-bd23-98f831b064d0",
"metadata": {},
"metadata": {
"editable": true,
"slideshow": {
"slide_type": ""
},
"tags": []
},
"outputs": [],
"source": [
"settings.auto_save_settings = False\n",
"settings.reset()"
"config.autosave = False\n",
"config.reset()"
]
},
{
Expand All @@ -87,7 +93,7 @@
"tags": []
},
"source": [
"The primary key to control the cache in the settings is :ref:`cache-policy <cache_policies>`. The default value is :ref:`\"off\" <off_cache_policy>`, which means that no caching is available. \n",
"The primary key to control the cache in the configuration is :ref:`cache-policy <cache_policies>`. The default value is :ref:`\"off\" <off_cache_policy>`, which means that no caching is available. \n",
"\n",
"In this case all files are downloaded into an **unmanaged** temporary directory created by *tempfile.TemporaryDirectory*. Since caching is disabled all calls to :func:`from_source` for remote services and URLSs will download the data again! This temporary directory will be unique for each earthkit-data session. When the directory object goes out of scope (at the latest on exit) the directory will be **cleaned up**. "
]
Expand All @@ -103,7 +109,7 @@
"tags": []
},
"source": [
"The settings tells us the current cache policy:"
"The config tells us the current cache policy:"
]
},
{
Expand All @@ -124,7 +130,7 @@
}
],
"source": [
"settings.get(\"cache-policy\")"
"config.get(\"cache-policy\")"
]
},
{
Expand Down Expand Up @@ -157,7 +163,7 @@
{
"data": {
"text/plain": [
"'/var/folders/93/w0p869rx17q98wxk83gn9ys40000gn/T/tmpuc3s5y8r'"
"'/var/folders/93/w0p869rx17q98wxk83gn9ys40000gn/T/tmptdj20reb'"
]
},
"execution_count": 4,
Expand All @@ -181,7 +187,7 @@
"tags": []
},
"source": [
"We can specify the parent directory for the temporary directory by using the :ref:`temporary-directory-root <off_cache_policy>` settings. By default it is set to None (no parent directory specified)."
"We can specify the parent directory for the temporary directory by using the :ref:`temporary-directory-root <off_cache_policy>` config option. By default it is set to None (no parent directory specified)."
]
},
{
Expand All @@ -199,7 +205,7 @@
{
"data": {
"text/plain": [
"'/var/folders/93/w0p869rx17q98wxk83gn9ys40000gn/T/tmpnjz5cnc_'"
"'/var/folders/93/w0p869rx17q98wxk83gn9ys40000gn/T/tmpfybjqiu6'"
]
},
"execution_count": 5,
Expand All @@ -210,14 +216,20 @@
"source": [
"s = {\"cache-policy\": \"off\", \n",
" \"temporary-directory-root\": \"~/my_demo_tmp\"}\n",
"settings.set(s)\n",
"config.set(s)\n",
"cache.directory()"
]
},
{
"cell_type": "markdown",
"id": "8d0ede62-77d4-4914-86c8-12bc6846d16a",
"metadata": {},
"metadata": {
"editable": true,
"slideshow": {
"slide_type": ""
},
"tags": []
},
"source": [
"#### Temporary cache directory"
]
Expand Down Expand Up @@ -259,8 +271,8 @@
}
],
"source": [
"settings.set(\"cache-policy\", \"temporary\")\n",
"print(settings.get(\"cache-policy\"))"
"config.set(\"cache-policy\", \"temporary\")\n",
"print(config.get(\"cache-policy\"))"
]
},
{
Expand Down Expand Up @@ -293,7 +305,7 @@
{
"data": {
"text/plain": [
"'/var/folders/93/w0p869rx17q98wxk83gn9ys40000gn/T/tmp43s9c97m'"
"'/var/folders/93/w0p869rx17q98wxk83gn9ys40000gn/T/tmp_i65c09a'"
]
},
"execution_count": 7,
Expand All @@ -318,7 +330,7 @@
"tags": []
},
"source": [
"We can specify the parent directory for the the temporary cache by using the :ref:`temporary-cache-directory-root <temporary_cache_policy>` settings. By default it is set to None (no parent directory specified)."
"We can specify the parent directory for the the temporary cache by using the :ref:`temporary-cache-directory-root <temporary_cache_policy>` config option. By default it is set to None (no parent directory specified)."
]
},
{
Expand All @@ -336,7 +348,7 @@
{
"data": {
"text/plain": [
"'~/my_demo_cache/tmp2jeqp8ig'"
"'/Users/cgr/my_demo_cache/tmp0yxt25rk'"
]
},
"execution_count": 8,
Expand All @@ -347,7 +359,7 @@
"source": [
"s = {\"cache-policy\": \"temporary\", \n",
" \"temporary-cache-directory-root\": \"~/my_demo_cache\"}\n",
"settings.set(s)\n",
"config.set(s)\n",
"cache.directory()"
]
},
Expand All @@ -372,7 +384,7 @@
"tags": []
},
"source": [
"When the :ref:`cache-policy <cache_policies>` is :ref:`\"user\" <user_cache_policy>` the **cache will be active** and created in a **managed directory** defined by the :ref:`user-cache-directory <user_cache_policy>` settings. \n",
"When the :ref:`cache-policy <cache_policies>` is :ref:`\"user\" <user_cache_policy>` the **cache will be active** and created in a **managed directory** defined by the :ref:`user-cache-directory <user_cache_policy>` config option. \n",
"\n",
"The user cache directory is **not cleaned up on exit**. So next time you start earthkit-data it will be there again unless it is deleted manually or it is set in way that on each startup a different path is assigned to it. Also, when you run multiple sessions of earthkit-data under the same user they will share the same cache. "
]
Expand All @@ -389,7 +401,7 @@
"tags": []
},
"source": [
"The settings tells us all the details about the cache policy and location:"
"The configuration tells us all the details about the cache policy and location:"
]
},
{
Expand All @@ -414,9 +426,9 @@
}
],
"source": [
"settings.set(\"cache-policy\", \"user\")\n",
"print(settings.get(\"cache-policy\"))\n",
"print(settings.get(\"user-cache-directory\"))"
"config.set(\"cache-policy\", \"user\")\n",
"print(config.get(\"cache-policy\"))\n",
"print(config.get(\"user-cache-directory\"))"
]
},
{
Expand Down Expand Up @@ -484,7 +496,7 @@
{
"data": {
"text/plain": [
"'~/earthkit-data-demo-cache'"
"'/Users/cgr/earthkit-data-demo-cache'"
]
},
"execution_count": 11,
Expand All @@ -493,7 +505,7 @@
}
],
"source": [
"settings.set(\"user-cache-directory\", \"~/earthkit-data-demo-cache\")\n",
"config.set(\"user-cache-directory\", \"~/earthkit-data-demo-cache\")\n",
"cache.directory()"
]
},
Expand All @@ -514,9 +526,9 @@
],
"metadata": {
"kernelspec": {
"display_name": "dev",
"display_name": "dev_ecc",
"language": "python",
"name": "dev"
"name": "dev_ecc"
},
"language_info": {
"codemirror_mode": {
Expand Down
Loading

0 comments on commit 4e2c14d

Please sign in to comment.