Merge pull request #151 from ONSdigital/BLAIS5-4235

Promote from BLAIS5-4235 to main

Author: kristian4res

Makefile

@@ -27,16 +27,3 @@ test: lint check-types
 
 requirements.txt:
 	@poetry export -f requirements.txt --without-hashes --output requirements.txt
-
-# TODO remove me
-.PHONY=deploy-log-error
-deploy-log-error: requirements.txt
-	 @gcloud functions deploy log-error \
-        --region=europe-west2 \
-        --entry-point=log_error \
-        --trigger-http \
-        --runtime=python39
-
-# TODO remove me
-.PHONY=deploy
-deploy: deploy-slack-alerts deploy-log-error

README.md

@@ -4,7 +4,7 @@ This is a Cloud Function which is used to send alerts from Google Cloud Logging
 
 ## How it Works
 
-Some GCP infrastructure is required for this to working.
+Some GCP infrastructure is required for this to work.
 
 1. **A PubSub Topic**:<br>
     Log messages will be put on this topic and trigger the Cloud Function.
@@ -50,30 +50,32 @@ This repository uses poetry. After cloning, install the dependencies by running:
 poetry install
 ```
 
+
 ### Makefile
 
 A `Makefile` is included with some useful tasks to help with development.
 Running `make help` will list all available commands.
 
-### GitHub Actions
-
-The GitHub Actions run the linter, typechecker and tests.
+### Linting and Testing
 
-To avoid getting failures, it's worth running `make test` before commit.
-Note, `make test` also runs the typechecker and linter. 
+The [GitHub Actions](https://docs.github.com/en/actions), a CI/CD platform, runs the linter, typechecker and tests (using workflows), whenever a GitHub PR is raised.
 
-### Linting Errors
+To minimise chances of failures when the GitHub Actions are ran, it's worth running `make test` before you push and commit to GitHub.
+Note, `make test` also runs the typechecker and linter.
 
 Linting errors can usually be fixed quickly with `make format`.
 
-### How to silence prod alerts 
+
+### How to silence specific event logs
+
 1. Navigate to the log entry in GCP Console and copy the entry to the clipboard
 2. Create a test in the `test_main.py` file using the copied log entry
 3. Run tests using `make format test` - the test you just created should fail!
 4. Navigate to the `lib/filters` dir and create a new `.py` file 
 5. Add new functionality to the newly created file (see `osconfig_agent_filter.py` for an example)
 6. Navigate to the `tests/lib/filters` dir and create a new `test_XX.py` file
 7. Create unit tests that test the actual filter functionality (again, check `test_osconfig_agent_filter.py` for an example). You will need to change the fixture!
+    - **NB** Event logs can be difficult to replicate in a sandbox, so it is important that the unit tests are present and accurately written before it is deployed to an environment.
 8. In `send_alerts.py`, import the function you just created and add it to the filter array `[]` in the `log_entry_skipped` function
 ```python
 def log_entry_skipped(log_entry: ProcessedLogEntry):
@@ -83,6 +85,6 @@ def log_entry_skipped(log_entry: ProcessedLogEntry):
         agent_connect_filter,
         ... etc]
 ```
-9. Run `make format test` - if all pass, push it up!
-
-**NB.** Slack alerts will only be visible in "dev", "dev-training", "preprod" and "prod"
+9. Run `make format test` - if the checks pass, push and commit!
+10. Deploy the Cloud Function in a sandbox and ensure it works as expected.
+    - **NB** Logs coming from sandboxes are filtered by default. If you want to reproduce error logs within a sandbox, make sure to remove `sandbox_filter` in `send_alerts/log_entry_skipped` before deploying the Cloud Function.

lib/cloud_run_revision/parse_event.py

@@ -13,18 +13,8 @@ def assert_field_in_event(field_name: str, event: dict) -> None:
         raise InvalidCloudRunRevisionEvent(f"Field '{field_name}' is missing.")
 
 
-def assert_is_v1_pubsub_message(event: dict) -> None:
-    if event["@type"] != "type.googleapis.com/google.pubsub.v1.PubsubMessage":
-        raise InvalidCloudRunRevisionEvent(
-            "Field '@type' is 'type.googleapis.com/google.pubsub.v1.PubsubMessage'. "
-            f"Got '{event['@type']}'"
-        )
-
-
 def parse_event(event) -> Event:
     assert_field_in_event("data", event)
-    assert_field_in_event("@type", event)
-    assert_is_v1_pubsub_message(event)
 
     try:
         return Event(data=json.loads(base64.b64decode(event["data"])))

poetry.lock

@@ -5,7 +5,7 @@ description = ""
 authors = ["Tom Oram <tom.oram@armakuni.com>"]
 
 [tool.poetry.dependencies]
-python = "^3.8"
+python = "^3.9"
 Flask = "^2.1.3"
 google-cloud-logging = "^3.2.1"
 requests = "^2.28.1"

pyproject.toml

@@ -19,23 +19,6 @@ def event():
     }
 
 
-def test_parse_event_fails_when_type_does_not_exist(event):
-    del event["@type"]
-    with pytest.raises(InvalidCloudRunRevisionEvent) as e:
-        parse_event(event)
-    assert e.value.args[0] == "Field '@type' is missing."
-
-
-def test_parse_event_fails_when_type_is_not_v1_pubsub_message(event):
-    event["@type"] = "unknown"
-    with pytest.raises(InvalidCloudRunRevisionEvent) as e:
-        parse_event(event)
-    assert (
-        e.value.args[0]
-        == "Field '@type' is 'type.googleapis.com/google.pubsub.v1.PubsubMessage'. Got 'unknown'"
-    )
-
-
 def test_parse_event_fails_when_data_does_not_exist(event):
     del event["data"]
     with pytest.raises(InvalidCloudRunRevisionEvent) as e: