Weather Query Improvement with Forecast Runtime Query (#621)

* Simplify Table Name Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * RAW query on Forecast Run Time Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Update Tests Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Update tests Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Update Tests Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Update Tests Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Refactor raw_parameters build Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Update Query Builder Table Connection Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Review to include options and refactor test Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * black formatting Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Remove 'source' parameter from WeatherQueryBuilder methods Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Examples Added to Documentation Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Remove duplication in tests Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Remove unused Variables and include timestamp coulmn name default Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Documentation and Examples Formatting Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Documentation Update Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> --------- Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com>
rtdip · Jan 11, 2024 · f769833 · f769833
1 parent d635f22
commit f769833
Show file tree

Hide file tree

Showing 10 changed files with 562 additions and 226 deletions.
diff --git a/docs/sdk/code-reference/query/functions/weather/latest.md b/docs/sdk/code-reference/query/functions/weather/latest.md
@@ -1,2 +1,61 @@
 # Weather Latest Function
-::: src.sdk.python.rtdip_sdk.queries.weather.latest
+::: src.sdk.python.rtdip_sdk.queries.weather.latest
+
+## Example get_point
+
+```python
+from rtdip_sdk.authentication.azure import DefaultAuth
+from rtdip_sdk.queries.weather.latest import get_point
+from rtdip_sdk.connectors import DatabricksSQLConnection
+
+auth = DefaultAuth().authenticate()
+token = auth.get_token("2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default").token
+connection = DatabricksSQLConnection("{server_hostname}", "{http_path}", token)
+
+params = {
+    "forecast": "mock_forecast",
+    "forecast_type": "mock_weather",
+    "region": "mock_region",
+    "data_security_level": "mock_security",
+    "data_type": "mock_data_type",
+    "lat": 1.1,
+    "lon": 1.1,
+}
+
+x = get_point(connection, params)
+
+print(x)
+```
+
+## Example get_grid
+
+```python
+from rtdip_sdk.authentication.azure import DefaultAuth
+from rtdip_sdk.queries.weather.latest import get_point
+from rtdip_sdk.connectors import DatabricksSQLConnection
+
+auth = DefaultAuth().authenticate()
+token = auth.get_token("2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default").token
+connection = DatabricksSQLConnection("{server_hostname}", "{http_path}", token)
+
+params = {
+    "forecast": "mock_forecast",
+    "forecast_type": "mock_weather",
+    "region": "mock_region",
+    "data_security_level": "mock_security",
+    "data_type": "mock_data_type",
+    "min_lat": 36,
+    "max_lat": 38,
+    "min_lon": -109.1,
+    "max_lon": -107.1,
+}
+
+x = get_grid(connection, params)
+
+print(x)
+```
+
+These examples are using [```DefaultAuth()```](../../../authentication/azure.md) and [```DatabricksSQLConnection()```](../../connectors/db-sql-connector.md) to authenticate and connect. You can find other ways to authenticate [here](../../../authentication/azure.md). The alternative built in connection methods are either by [```PYODBCSQLConnection()```](../../connectors/pyodbc-sql-connector.md), [```TURBODBCSQLConnection()```](../../connectors/turbodbc-sql-connector.md) or [```SparkConnection()```](../../connectors/spark-connector.md).
+
+!!! note "Note"
+    </b>```server_hostname``` and ```http_path``` can be found on the [SQL Warehouses Page](../../../../queries/databricks/sql-warehouses.md). <br />
diff --git a/docs/sdk/code-reference/query/functions/weather/raw.md b/docs/sdk/code-reference/query/functions/weather/raw.md
@@ -1,2 +1,73 @@
 # Weather Raw Function
-::: src.sdk.python.rtdip_sdk.queries.weather.raw
+::: src.sdk.python.rtdip_sdk.queries.weather.raw
+
+## Example get_point
+
+```python
+from rtdip_sdk.authentication.azure import DefaultAuth
+from rtdip_sdk.queries.weather.raw import get_point
+from rtdip_sdk.connectors import DatabricksSQLConnection
+
+auth = DefaultAuth().authenticate()
+token = auth.get_token("2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default").token
+connection = DatabricksSQLConnection("{server_hostname}", "{http_path}", token)
+
+params = {
+    "forecast": "mock_forecast",
+    "forecast_type": "mock_weather",
+    "region": "mock_region",
+    "data_security_level": "mock_security",
+    "data_type": "mock_data_type",
+    "lat": 1.1,
+    "lon": 1.1,
+    "start_date": "2020-01-01",
+    "end_date": "2020-01-02",
+    "forecast_run_start_date": "2020-01-01",
+    "forecast_run_end_date": "2020-01-02",
+    "timestamp_column": "EventTime",
+    "forecast_run_timestamp_column": "EnqueuedTime",
+}
+
+x = get_point(connection, params)
+
+print(x)
+```
+
+## Example get_grid
+
+```python
+from rtdip_sdk.authentication.azure import DefaultAuth
+from rtdip_sdk.queries.weather.raw import get_grid
+from rtdip_sdk.connectors import DatabricksSQLConnection
+
+auth = DefaultAuth().authenticate()
+token = auth.get_token("2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default").token
+connection = DatabricksSQLConnection("{server_hostname}", "{http_path}", token)
+
+params = {
+    "forecast": "mock_forecast",
+    "forecast_type": "mock_weather",
+    "region": "mock_region",
+    "data_security_level": "mock_security",
+    "data_type": "mock_data_type",
+    "min_lat": 36,
+    "max_lat": 38,
+    "min_lon": -109.1,
+    "max_lon": -107.1,
+    "start_date": "2020-01-01",
+    "end_date": "2020-01-02",
+    "forecast_run_start_date": "2020-01-01",
+    "forecast_run_end_date": "2020-01-02",
+    "timestamp_column": "EventTime",
+    "forecast_run_timestamp_column": "EnqueuedTime",
+}
+
+x = get_grid(connection, params)
+
+print(x)
+```
+
+These examples are using [```DefaultAuth()```](../../../authentication/azure.md) and [```DatabricksSQLConnection()```](../../connectors/db-sql-connector.md) to authenticate and connect. You can find other ways to authenticate [here](../../../authentication/azure.md). The alternative built in connection methods are either by [```PYODBCSQLConnection()```](../../connectors/pyodbc-sql-connector.md), [```TURBODBCSQLConnection()```](../../connectors/turbodbc-sql-connector.md) or [```SparkConnection()```](../../connectors/spark-connector.md).
+
+!!! note "Note"
+    </b>```server_hostname``` and ```http_path``` can be found on the [SQL Warehouses Page](../../../../queries/databricks/sql-warehouses.md). <br />
diff --git a/docs/sdk/code-reference/query/functions/weather/weather_query_builder.md b/docs/sdk/code-reference/query/functions/weather/weather_query_builder.md
@@ -1,2 +1,8 @@
 # Weather Query Builder
-::: src.sdk.python.rtdip_sdk.queries.weather.weather_query_builder
+::: src.sdk.python.rtdip_sdk.queries.weather.weather_query_builder
+
+!!! note "Note"
+    </b>These examples are using [```DefaultAuth()```](../../../authentication/azure.md) and [```DatabricksSQLConnection()```](../../connectors/db-sql-connector.md) to authenticate and connect. You can find other ways to authenticate [here](../../../authentication/azure.md). The alternative built in connection methods are either by [```PYODBCSQLConnection()```](../../connectors/pyodbc-sql-connector.md), [```TURBODBCSQLConnection()```](../../connectors/turbodbc-sql-connector.md) or [```SparkConnection()```](../../connectors/spark-connector.md).<br />
+
+!!! note "Note"
+    </b>```server_hostname``` and ```http_path``` can be found on the [SQL Warehouses Page](../../../../queries/databricks/sql-warehouses.md). <br />
diff --git a/src/sdk/python/rtdip_sdk/queries/weather/_weather_query_builder.py b/src/sdk/python/rtdip_sdk/queries/weather/_weather_query_builder.py
@@ -32,51 +32,45 @@ def _build_parameters(
     area_type: str,
     table_type: str,
 ) -> dict:
-    if area_type == "grid":
-        raw_parameters = {
-            "forecast": parameters_dict.get("forecast", None),
-            "region": parameters_dict.get("region"),
-            "data_security_level": parameters_dict.get("data_security_level"),
-            "data_type": parameters_dict.get("data_type"),
-            "start_date": parameters_dict["start_date"],
-            "end_date": parameters_dict["end_date"],
-            "max_lat": parameters_dict["max_lat"],
-            "max_lon": parameters_dict["max_lon"],
-            "min_lat": parameters_dict["min_lat"],
-            "min_lon": parameters_dict["min_lon"],
-            "source": parameters_dict.get("source", None),
-            "limit": parameters_dict.get("limit", None),
-            "latitude_column": parameters_dict.get("latitude_column", "Latitude"),
-            "longitude_column": parameters_dict.get("longitude_column", "Longitude"),
-            "tagname_column": parameters_dict.get("tagname_column", "TagName"),
-        }
-        if table_type == "raw":
-            raw_parameters["timestamp_column"] = parameters_dict.get(
-                "timestamp_column", "EventTime"
-            )
-            raw_parameters["include_status"] = False
+    raw_parameters = {
+        "source": parameters_dict.get("source", None),
+        "forecast": parameters_dict.get("forecast", None),
+        "forecast_type": parameters_dict.get("forecast_type", "weather"),
+        "region": parameters_dict.get("region", None),
+        "data_security_level": parameters_dict.get("data_security_level", None),
+        "data_type": parameters_dict.get("data_type", None),
+        "limit": parameters_dict.get("limit", None),
+        "latitude_column": parameters_dict.get("latitude_column", "Latitude"),
+        "longitude_column": parameters_dict.get("longitude_column", "Longitude"),
+        "tagname_column": parameters_dict.get("tagname_column", "TagName"),
+    }
 
     if area_type == "point":
-        raw_parameters = {
-            "forecast": parameters_dict.get("forecast", None),
-            "region": parameters_dict.get("region"),
-            "data_security_level": parameters_dict.get("data_security_level"),
-            "data_type": parameters_dict.get("data_type"),
-            "start_date": parameters_dict["start_date"],
-            "end_date": parameters_dict["end_date"],
-            "lat": parameters_dict["lat"],
-            "lon": parameters_dict["lon"],
-            "source": parameters_dict.get("source", None),
-            "limit": parameters_dict.get("limit", None),
-            "latitude_column": parameters_dict.get("latitude_column", "Latitude"),
-            "longitude_column": parameters_dict.get("longitude_column", "Longitude"),
-            "tagname_column": parameters_dict.get("tagname_column", "TagName"),
-        }
-        if table_type == "raw":
-            raw_parameters["timestamp_column"] = parameters_dict.get(
-                "timestamp_column", "EventTime"
-            )
-            raw_parameters["include_status"] = False
+        raw_parameters["lat"] = parameters_dict["lat"]
+        raw_parameters["lon"] = parameters_dict["lon"]
+
+    if area_type == "grid":
+        raw_parameters["max_lat"] = parameters_dict["max_lat"]
+        raw_parameters["max_lon"] = parameters_dict["max_lon"]
+        raw_parameters["min_lat"] = parameters_dict["min_lat"]
+        raw_parameters["min_lon"] = parameters_dict["min_lon"]
+
+    if table_type == "raw":
+        raw_parameters["start_date"] = parameters_dict["start_date"]
+        raw_parameters["end_date"] = parameters_dict["end_date"]
+        raw_parameters["forecast_run_start_date"] = parameters_dict[
+            "forecast_run_start_date"
+        ]
+        raw_parameters["forecast_run_end_date"] = parameters_dict[
+            "forecast_run_end_date"
+        ]
+        raw_parameters["timestamp_column"] = parameters_dict.get(
+            "timestamp_column", "EventTime"
+        )
+        raw_parameters["forecast_run_timestamp_column"] = parameters_dict.get(
+            "forecast_run_timestamp_column", "EnqueuedTime"
+        )
+        raw_parameters["include_status"] = False
 
     return raw_parameters
 
@@ -87,16 +81,14 @@ def _raw_query_grid(parameters_dict: dict) -> str:
         "{% if source is defined and source is not none %}"
         "`{{ source|lower }}` "
         "{% else %}"
-        "`{{ forecast|lower }}`.`weather`.`{{ region|lower }}_weather_{{ data_security_level|lower }}_events_{{ data_type|lower }}` "
-        "{% endif %}"
-        'WHERE `{{ timestamp_column }}` BETWEEN to_timestamp("{{ start_date }}") AND to_timestamp("{{ end_date }}")'
-        "AND `{{ latitude_column }}` > '{{ min_lat}}' "
-        "AND `{{ latitude_column }}` < '{{ max_lat}}' "
-        "AND `{{ longitude_column }}` > '{{ min_lon}}' "
-        "AND`{{ longitude_column }}` < '{{ max_lon}}' "
-        "{% if source is defined and source is not none %}"
-        "AND SOURCE = '{{ source }}' "
+        "`{{ forecast|lower }}`.`{{ forecast_type|lower }}`.`{{ region|lower }}_{{ data_security_level|lower }}_events_{{ data_type|lower }}` "
         "{% endif %}"
+        'WHERE (`{{ timestamp_column }}` BETWEEN to_timestamp("{{ start_date }}") AND to_timestamp("{{ end_date }}")) '
+        'AND (`{{ forecast_run_timestamp_column }}` BETWEEN to_timestamp("{{ forecast_run_start_date }}") AND to_timestamp("{{ forecast_run_end_date }}")) '
+        "AND `{{ latitude_column }}` > {{ min_lat}} "
+        "AND `{{ latitude_column }}` < {{ max_lat}} "
+        "AND `{{ longitude_column }}` > {{ min_lon}} "
+        "AND `{{ longitude_column }}` < {{ max_lon}} "
         "ORDER BY `{{ tagname_column }}` "
         "{% if limit is defined and limit is not none %}"
         "LIMIT {{ limit }} "
@@ -115,14 +107,12 @@ def _raw_query_point(parameters_dict: dict) -> str:
         "{% if source is defined and source is not none %}"
         "`{{ source|lower }}` "
         "{% else %}"
-        "`{{ forecast|lower }}`.`weather`.`{{ region|lower }}_weather_{{ data_security_level|lower }}_events_{{ data_type|lower }}` "
-        "{% endif %}"
-        'WHERE `{{ timestamp_column }}` BETWEEN to_timestamp("{{ start_date }}") AND to_timestamp("{{ end_date }}")'
-        "AND `{{ latitude_column }}` > '{{lat}}' "
-        "AND `{{ longitude_column }}` > '{{lon}}' "
-        "{% if source is defined and source is not none %}"
-        "AND SOURCE = '{{ source }}' "
+        "`{{ forecast|lower }}`.`{{ forecast_type|lower }}`.`{{ region|lower }}_{{ data_security_level|lower }}_events_{{ data_type|lower }}` "
         "{% endif %}"
+        'WHERE (`{{ timestamp_column }}` BETWEEN to_timestamp("{{ start_date }}") AND to_timestamp("{{ end_date }}")) '
+        'AND (`{{ forecast_run_timestamp_column }}` BETWEEN to_timestamp("{{ forecast_run_start_date }}") AND to_timestamp("{{ forecast_run_end_date }}")) '
+        "AND `{{ latitude_column }}` == {{ lat }} "
+        "AND `{{ longitude_column }}` == {{ lon }} "
         "ORDER BY `{{ tagname_column }}` "
         "{% if limit is defined and limit is not none %}"
         "LIMIT {{ limit }} "
@@ -141,15 +131,12 @@ def _latest_query_grid(parameters_dict: dict) -> str:
         "{% if source is defined and source is not none %}"
         "`{{ source|lower }}` "
         "{% else %}"
-        "`{{ forecast|lower }}`.`weather`.`{{ region|lower }}_weather_{{ data_security_level|lower }}_events_{{ data_type|lower }}` "
-        "{% endif %}"
-        "WHERE `{{ latitude_column }}` > '{{ min_lat}}' "
-        "AND `{{ latitude_column }}` < '{{ max_lat}}' "
-        "AND `{{ longitude_column }}` > '{{ min_lon}}' "
-        "AND`{{ longitude_column }}` < '{{ max_lon}}' "
-        "{% if source is defined and source is not none %}"
-        "AND SOURCE = '{{ source }}' "
+        "`{{ forecast|lower }}`.`{{ forecast_type|lower }}`.`{{ region|lower }}_{{ data_security_level|lower }}_events_{{ data_type|lower }}_latest` "
         "{% endif %}"
+        "WHERE `{{ latitude_column }}` > {{ min_lat}} "
+        "AND `{{ latitude_column }}` < {{ max_lat}} "
+        "AND `{{ longitude_column }}` > {{ min_lon}} "
+        "AND `{{ longitude_column }}` < {{ max_lon}} "
         "ORDER BY `{{ tagname_column }}` "
         "{% if limit is defined and limit is not none %}"
         "LIMIT {{ limit }} "
@@ -168,13 +155,10 @@ def _latest_query_point(parameters_dict: dict) -> str:
         "{% if source is defined and source is not none %}"
         "`{{ source|lower }}` "
         "{% else %}"
-        "`{{ forecast|lower }}`.`weather`.`{{ region|lower }}_weather_{{ data_security_level|lower }}_events_{{ data_type|lower }}` "
-        "{% endif %}"
-        "WHERE `{{ latitude_column }}` == '{{lat}}' "
-        "AND `{{ longitude_column }}` == '{{lon}}' "
-        "{% if source is defined and source is not none %}"
-        "AND SOURCE = '{{ source }}' "
+        "`{{ forecast|lower }}`.`{{ forecast_type|lower }}`.`{{ region|lower }}_{{ data_security_level|lower }}_events_{{ data_type|lower }}_latest` "
         "{% endif %}"
+        "WHERE `{{ latitude_column }}` == {{ lat }} "
+        "AND `{{ longitude_column }}` == {{ lon }} "
         "ORDER BY `{{ tagname_column }}` "
         "{% if limit is defined and limit is not none %}"
         "LIMIT {{ limit }} "

diff --git a/src/sdk/python/rtdip_sdk/queries/weather/latest.py b/src/sdk/python/rtdip_sdk/queries/weather/latest.py
@@ -34,15 +34,16 @@ def get_grid(connection: object, parameters_dict: dict) -> pd.DataFrame:
         parameters_dict: A dictionary of parameters (see Attributes table below)
 
     Attributes:
-        forecast (str): Business unit
+        source (optional str): Source of the data the full table name
+        forecast (str): Any specific identifier for forecast
+        forecast_type(str): Type of forecast ie weather, solar, power, etc
         region (str): Region
         data_security_level (str): Level of data security
         data_type (str): Type of the data (float, integer, double, string)
         max_lat (float): Maximum latitude
         max_lon (float): Maximum longitude
         min_lat (float): Minimum latitude
         min_lon (float): Minimum longitude
-        source (optional str): Source of the data ie ECMWF
         limit (optional int): The number of rows to be returned
 
     Returns:
@@ -84,13 +85,14 @@ def get_point(connection: object, parameters_dict: dict) -> pd.DataFrame:
         parameters_dict: A dictionary of parameters (see Attributes table below)
 
     Attributes:
-        forecast (str): Business unit
+        source (optional str): Source of the data the full table name
+        forecast (str): Any specific identifier for forecast
+        forecast_type(str): Type of forecast ie weather, solar, power, etc
         region (str): Region
         data_security_level (str): Level of data security
         data_type (str): Type of the data (float, integer, double, string)
         lat (float): latitude
         lon (float): longitude
-        source (optional str): Source of the data ie ECMWF
         limit (optional int): The number of rows to be returned
 
     Returns: