Enable using url source as a stream

docs/examples.rst

@@ -35,6 +35,7 @@ Here is a list of example notebooks to illustrate how to use earthkit-data.
     examples/grib_file_pattern.ipynb
     examples/grib_tar.ipynb
     examples/grib_url.ipynb
+    examples/grib_url_stream.ipynb
     examples/grib_to_netcdf.ipynb
     examples/numpy_fieldlist.ipynb
     examples/grib_nearest_gridpoint.ipynb

docs/examples/fdb.ipynb

@@ -161,7 +161,7 @@
    "id": "c2c8809e-9d53-4c69-af57-e3617e1df8b8",
    "metadata": {},
    "source": [
-    "When we use the **group_by** option `from_source()` gives us a stream iterator object. Each iteration step results in a Fieldlist object, which is built by consuming GRIB messages from the stream until the values of the metadata keys specified in *group_by* change. The generated Fieldlist keeps GRIB messages in memory then gets deleted when going out of scope."
+    "When we use the [group_by](../guide/sources.rst#fdb) option [from_source()](../guide/sources.rst#fdb) gives us a stream iterator object. Each iteration step results in a Fieldlist object, which is built by consuming GRIB messages from the stream until the values of the metadata keys specified in [group_by](../guide/sources.rst#fdb) change. The generated Fieldlist keeps GRIB messages in memory then gets deleted when going out of scope."
    ]
   },
   {
@@ -206,7 +206,7 @@
    "id": "standard-soviet",
    "metadata": {},
    "source": [
-    "We can read multiple fields into memory from the stream at a time by using **batch_size** in *from_source()*, Please note that *batch_size* cannot be used together with *group_by*."
+    "We can read multiple fields into memory from the stream at a time by using [batch_size](../guide/sources.rst#fdb) in [from_source()](../guide/sources.rst#fdb), Please note that *batch_size* cannot be used together with [group_by](../guide/sources.rst#fdb)."
    ]
   },
   {
@@ -252,7 +252,7 @@
    "id": "personal-sense",
    "metadata": {},
    "source": [
-    "When we use **batch_size=0** all the fields are loaded into memory and the resulting object iswill behave like a FieldList:"
+    "When we use [batch_size](../guide/sources.rst#fdb)=0 all the fields are loaded into memory and the resulting object iswill behave like a FieldList:"
    ]
   },
   {
@@ -1202,9 +1202,9 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "mpy38",
+   "display_name": "dev",
    "language": "python",
-   "name": "mpy38"
+   "name": "dev"
   },
   "language_info": {
    "codemirror_mode": {
@@ -1216,7 +1216,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.16"
+   "version": "3.10.13"
   }
  },
  "nbformat": 4,

docs/examples/grib_fdb_write.ipynb

@@ -308,9 +308,9 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "pyfdb",
+   "display_name": "dev",
    "language": "python",
-   "name": "pyfdb"
+   "name": "dev"
   },
   "language_info": {
    "codemirror_mode": {
@@ -322,7 +322,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.12"
+   "version": "3.10.13"
   }
  },
  "nbformat": 4,

docs/examples/grib_url.ipynb

@@ -894,9 +894,9 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "mpy38",
+   "display_name": "dev",
    "language": "python",
-   "name": "mpy38"
+   "name": "dev"
   },
   "language_info": {
    "codemirror_mode": {
@@ -908,7 +908,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.16"
+   "version": "3.10.13"
   }
  },
  "nbformat": 4,

docs/examples/grib_url_stream.ipynb

@@ -0,0 +1,186 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "ca9aa5f2-705a-4d80-b400-6acf7ff86786",
+   "metadata": {},
+   "source": [
+    "## Reading GRIB from an URL as a stream"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "c23ffb07-747c-4535-b608-66b080d6d4e5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import earthkit.data"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0a58377e-7be5-4951-8a76-0d5c24444898",
+   "metadata": {},
+   "source": [
+    "earthkit-data can read GRIB data from a URL as a stream without writing anything to disk. This can be activated with the **stream=True** kwarg when calling [from_source()](../guide/sources.rst#url) with a \"url\" source.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "1d2b07e3-1820-45ec-88d5-c78765533735",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ds = earthkit.data.from_source(\"url\", \n",
+    "                       \"https://get.ecmwf.int/repository/test-data/earthkit-data/examples/test4.grib\", \n",
+    "                        stream=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "51964579-bf3f-4948-918d-fcd35581950e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "for f in ds:\n",
+    "    # f is GribField object. It gets deleted when going out of scope\n",
+    "    print(f)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7f595ec8-008f-4198-8a3a-f8346fbfe09c",
+   "metadata": {},
+   "source": [
+    "### Stream options"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "994802cc-2e74-4af4-8e20-031f5937df62",
+   "metadata": {},
+   "source": [
+    "To control how the stream is read use [batch_size](../guide/sources.rst#url) and [group_by](../guide/sources.rst#url). E.g. the following code reads the GRIB data in messages of 2."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "b7c9a685-6b38-4236-ae74-a14445e8bb95",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "len=2\n",
+      " GribField(t,500,20070101,1200,0,0)\n",
+      " GribField(z,500,20070101,1200,0,0)\n",
+      "len=2\n",
+      " GribField(t,850,20070101,1200,0,0)\n",
+      " GribField(z,850,20070101,1200,0,0)\n"
+     ]
+    }
+   ],
+   "source": [
+    "ds = earthkit.data.from_source(\"url\", \n",
+    "                       \"https://get.ecmwf.int/repository/test-data/earthkit-data/examples/test4.grib\", \n",
+    "                        stream=True, batch_size=2)\n",
+    "\n",
+    "for f in ds:\n",
+    "    # f is a fieldlist\n",
+    "    print(f\"len={len(f)}\")\n",
+    "    for g in f:\n",
+    "        print(f\" {g}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cb7d74b5-73e0-4e57-9d80-e45a9dc725d8",
+   "metadata": {},
+   "source": [
+    "### Multiple URLs"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6d5efce5-c9d0-4edf-a2cf-312ced9339b5",
+   "metadata": {},
+   "source": [
+    "The stream option works even when the input is a list of URLs."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "891af3e5-5aa5-451f-b7bb-a86eaf7c8ffe",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "len=3\n",
+      " GribField(t,500,20070101,1200,0,0)\n",
+      " GribField(z,500,20070101,1200,0,0)\n",
+      " GribField(t,850,20070101,1200,0,0)\n",
+      "len=3\n",
+      " GribField(z,850,20070101,1200,0,0)\n",
+      " GribField(t,1000,20180801,1200,0,0)\n",
+      " GribField(u,1000,20180801,1200,0,0)\n",
+      "len=3\n",
+      " GribField(v,1000,20180801,1200,0,0)\n",
+      " GribField(t,850,20180801,1200,0,0)\n",
+      " GribField(u,850,20180801,1200,0,0)\n",
+      "len=1\n",
+      " GribField(v,850,20180801,1200,0,0)\n"
+     ]
+    }
+   ],
+   "source": [
+    "ds = earthkit.data.from_source(\"url\", \n",
+    "                       [\"https://get.ecmwf.int/repository/test-data/earthkit-data/examples/test4.grib\", \n",
+    "                       \"https://get.ecmwf.int/repository/test-data/earthkit-data/examples/test6.grib\"], \n",
+    "                        stream=True, batch_size=3)\n",
+    "\n",
+    "for f in ds:\n",
+    "    # f is a fieldlist\n",
+    "    print(f\"len={len(f)}\")\n",
+    "    for g in f:\n",
+    "        print(f\" {g}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d287fb70-66fb-4b2f-8c07-a3cca7c3035d",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "dev",
+   "language": "python",
+   "name": "dev"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}