Feature/brownie docs (#27)

* feat: Document Brownie ns cleaner.
kindredgroup · May 1, 2024 · 91be587 · 91be587
1 parent 4009679
commit 91be587
Show file tree

Hide file tree

Showing 3 changed files with 111 additions and 4 deletions.
diff --git a/brownie/README.md b/brownie/README.md
@@ -2,24 +2,47 @@
 
 ## Introduction
 
-Brownie is a housekeeping daemon whose responsibility is to find and remove stale resources created by PIT but could not be cleaned for technical reasons (PIT test has stopped by user, or process existed unexpectedly etc.). About the name: https://en.wikipedia.org/wiki/Brownie_(folklore)
+Brownie is a housekeeping daemon whose responsibility is to find and remove stale resources created by PIT but could not be cleaned for technical reasons (PIT test has stopped by user, or process exited unexpectedly etc.). About the name: https://en.wikipedia.org/wiki/Brownie_(folklore)
 
-Brownie app is deployed into k8s cluster as periodic CronJob. It scans for resources matching a pre-determined naming pattern and removes them when they get old. Currently we support *PostgresSQL databases* and *Kafka topics*. More resource types may be added in the future.
+Brownie app is made up of two modules.
+- One module is intended to run inside K8s cluster. It takes care of infrastructure resources which are external to kubernetes such as old databases and topics.
+- Another module is intended to run outside of K8s cluster. It takes care of resources which are internal to kubernetes such as old namespaces and their internal objects.
+
+The following content is split into two sections explaining the details of each of these modules.
+
+# The Brownie App module
+
+Brownie app is the module which is intended to run inside K8s cluster. It is deployed into k8s cluster as periodic CronJob. Job scans for resources matching a pre-determined naming pattern and removes them when they get old. Currently we support *PostgresSQL databases* and *Kafka topics*. More resource types may be added in the future.
 
 Application needs to be configured with:
 - The access to PostgreSQL and Kafka using elevated privileges.
 - The regexp pattern. Brownie will use it to extract the resource creation timestamp from resource name and then use that timestamp for determining whether the resources should be removed.
 - The retention period of old resources. Resources younger than the retention period will not be deleted.
 
+The entry-point into this module is the main index file: `src/index.ts`
+
+
+## Scheduler
+
+Brownie app does not have scheduling logic. Its internal logic is designed to focus only on the cleaning functionality. The scheduling activities, such as time tracking, starting, stopping, pausing etc. are not part of Brownie app. Brownie relies on external scheduler. The good reliable scheduler is already availabe in K8s - "CronJobs" https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/
+
+When K8s schedules a job run it will create a fresh Brownie pod. The pod will exit once Brownie app completes its cleaning routine.
+
+## Deployment
+
+This module is packaged as docker image and published into Container Registry of GitHib Packages: https://docs.github.com/en/packages/learn-github-packages/introduction-to-github-packages. It should be deployed as k8s CronJob.
+
 ## Deployment Architecture
 
 ![](./docs/arch.png)
 
 ## How it works
 
+This section explains the details of what happens inside Brownie app when pod runs.
+
 Brownie has two internal logical modules with similar functionality. Each module connects to a dedicated resource server and scans for all available resources. For example, one such module is for monitoring a PostgreSQL server. Brownie connects to PostgreSQL server, fetches the list of all databases using name matching query `SELECT ... FROM ... WHERE dbname LIKE %pit%` and evaluates each found database. If database name contains a predefined timestamp pattern, then Brownie will either leave it as is or drop the database. The decision is based on database age. The database should be old enough and that is determined by _retention period_. The _retention period_ is a configuration parameter.
 
-The exaclty same logic applies to Kafka topics.
+The exactly same logic applies to Kafka topics.
 
 It goes without saying that we do not want to drop resources which are currently being used or which are not used now but will be used soon. The key aspect of Brownie functionality is its ability to determine the age of resource. Theoretically, the most straightforward way to find out when resource was created is to query the resource metadata. However, in practice this is not so simple and might not be possible for some resources. As scanning resource metadata is somewhat very specific to resource type and may also depend on the specific version of resource servers (PosgreSQL, Kafka, ElasticSearch and etc). To provide a generic and simple solution we ask that ephemeral resources, such as temporary databases, topics (and other resources which we will support in the future) **have creation timestamp built into the resource name**.
 
@@ -55,4 +78,47 @@ Brownie does not enforce any specific naming strategy for your resources other t
 | --kafka-password       | KAFKA_PASSWORD       |
 | --kafka-sasl-mechanism | KAFKA_SASL_MECHANISM | Optional
 
+# The external job module
+
+The external job module is intended to run outside of K8s cluster. It is made of two packages:
+- the external scheduler (the launcher)
+- the node application.
+
+Please refer to the architecture diagram.
+
+Sources for the node application package can be found under `src/k8ns/`. Hence, the entry-point is `src/k8ns/index.ts` The purpose of this package is to delete old sub-namespaces from K8s.
+
+It requires two inputs:
+- The retention period of old namespaces. Namespaces younger than the retention period will not be deleted.
+- The JSON file containing the list of objects each describing a child namespace.
+
+## Scheduler
+
+The job scheduling is delegated to CI/CD. Any CI/CD solution like Jenkins and similar, can be used if it can invoke a shell script. It is expected that CI will periodically invoke this script: `brownie/scripts/clean-ns.sh`
 
+## Deployment
+
+The good news is that there is nothing to be deployed. All what is needed is to configure CI to host a periodic Job which can checkout brownie code and invoke a shell command.
+
+## Deployment Architecture
+
+![](./docs/arch-external-job.png)
+
+
+## How it works
+
+Scheduler will invoke a shell script which then query K8s cluster, generate a list of namespaces in JSON format and write into the file. Then script will launch a node process passing the file as parameter. This design allows us to keep node app simple and small. The code boils down to parsing the JSON, extracting old namespaces from the given list, and deleting them one by one. The deletion logic is re-used from PIT K8s-deployer. Since k8s-deployer can create and clean namespaces, we can re-use its cleaning code. Here is the implementation: `pit-toolkit/k8s-deployer/scripts/k8s-manage-namespaces.sh`
+
+The conceptual algorithm looks like this:
+
+1. CI job awakes and invokes `clean-ns.sh`;
+2. The `clean-ns.sh` queries K8s and makes JSON file;
+3. The `clean-ns.sh` launches node app;
+4. The node app cleans old namespaces.
+
+## Configuration
+
+| Parameter              | Env variable         | Description
+|------------------------|----------------------|-------------------
+| --dry-run              | DRY_RUN              | "true" or "false". When "true" resources will not be deleted
+| --retention-period     | RETENTION_PERIOD     | 1day, Ndays, 1hour, Nhours, 1minute, Nminutes and etc.
diff --git a/brownie/docs/arch-external-job.png b/brownie/docs/arch-external-job.png
diff --git a/brownie/docs/arch.drawio b/brownie/docs/arch.drawio
@@ -1,4 +1,4 @@
-<mxfile host="Electron" modified="2024-04-17T01:21:46.532Z" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/21.2.8 Chrome/112.0.5615.165 Electron/24.2.0 Safari/537.36" etag="ULA6Qt2QZjMX24y8XgBv" version="21.2.8" type="device">
+<mxfile host="Electron" modified="2024-04-30T04:27:27.939Z" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/21.2.8 Chrome/112.0.5615.165 Electron/24.2.0 Safari/537.36" etag="sdj7iLuGJj_pdFVgPH1K" version="21.2.8" type="device" pages="2">
   <diagram name="Page-1" id="4C_ySlymoL6B5oeubTcv">
     <mxGraphModel dx="1752" dy="732" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="827" pageHeight="1169" math="0" shadow="0">
       <root>
@@ -136,4 +136,45 @@
       </root>
     </mxGraphModel>
   </diagram>
+  <diagram id="vfUj89OVI38U2ls6i8lU" name="Page-2">
+    <mxGraphModel dx="925" dy="732" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="827" pageHeight="1169" math="0" shadow="0">
+      <root>
+        <mxCell id="0" />
+        <mxCell id="1" parent="0" />
+        <mxCell id="eHQvnrRlQ7p6zjHkYq-G-1" value="K8s Cluster" style="rounded=0;whiteSpace=wrap;html=1;dashed=1;strokeColor=#B3B3B3;fontColor=#FFFFFF;verticalAlign=top;" vertex="1" parent="1">
+          <mxGeometry x="400" y="80" width="310" height="370" as="geometry" />
+        </mxCell>
+        <mxCell id="Eu_Z3l2dlq1L3zP6mrAY-2" value="pit-sub-ns-1" style="rounded=0;whiteSpace=wrap;html=1;verticalAlign=top;dashed=1;" vertex="1" parent="1">
+          <mxGeometry x="454" y="130" width="200" height="120" as="geometry" />
+        </mxCell>
+        <mxCell id="Eu_Z3l2dlq1L3zP6mrAY-7" value="pit-sub-ns-1" style="rounded=0;whiteSpace=wrap;html=1;verticalAlign=top;dashed=1;" vertex="1" parent="1">
+          <mxGeometry x="455" y="290" width="200" height="120" as="geometry" />
+        </mxCell>
+        <mxCell id="Eu_Z3l2dlq1L3zP6mrAY-8" value="CI (Jenkins or other)" style="rounded=0;whiteSpace=wrap;html=1;dashed=1;strokeColor=#B3B3B3;fontColor=#FFFFFF;verticalAlign=top;" vertex="1" parent="1">
+          <mxGeometry x="20" y="80" width="280" height="370" as="geometry" />
+        </mxCell>
+        <mxCell id="Eu_Z3l2dlq1L3zP6mrAY-9" value="Periodic Job" style="rounded=0;whiteSpace=wrap;html=1;verticalAlign=top;dashed=1;" vertex="1" parent="1">
+          <mxGeometry x="50" y="140" width="220" height="260" as="geometry" />
+        </mxCell>
+        <mxCell id="Eu_Z3l2dlq1L3zP6mrAY-10" value="&lt;div style=&quot;text-align: left;&quot;&gt;&lt;span style=&quot;background-color: initial;&quot;&gt;/pit-toolkit/&lt;/span&gt;&lt;/div&gt;&lt;div style=&quot;text-align: left;&quot;&gt;&lt;span style=&quot;background-color: initial;&quot;&gt;&amp;nbsp; &amp;nbsp;/brownie/&lt;/span&gt;&lt;/div&gt;&lt;div style=&quot;text-align: left;&quot;&gt;&lt;span style=&quot;background-color: initial;&quot;&gt;&amp;nbsp; &amp;nbsp;/k8s-deployer/&lt;/span&gt;&lt;/div&gt;" style="rounded=0;whiteSpace=wrap;html=1;verticalAlign=top;fontFamily=Courier New;strokeColor=#FF8000;strokeWidth=2;" vertex="1" parent="1">
+          <mxGeometry x="80" y="190" width="160" height="70" as="geometry" />
+        </mxCell>
+        <mxCell id="TMhfgzIf1tkrQ-7BzbZ0-1" style="edgeStyle=none;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;entryX=0;entryY=0.5;entryDx=0;entryDy=0;strokeColor=#FF8000;exitX=1;exitY=0.5;exitDx=0;exitDy=0;" edge="1" parent="1" source="Eu_Z3l2dlq1L3zP6mrAY-10" target="Eu_Z3l2dlq1L3zP6mrAY-2">
+          <mxGeometry relative="1" as="geometry">
+            <mxPoint x="740" y="256" as="sourcePoint" />
+            <mxPoint x="921" y="256" as="targetPoint" />
+          </mxGeometry>
+        </mxCell>
+        <mxCell id="TMhfgzIf1tkrQ-7BzbZ0-2" style="edgeStyle=none;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;entryX=0;entryY=0.5;entryDx=0;entryDy=0;strokeColor=#FF8000;exitX=1;exitY=0.5;exitDx=0;exitDy=0;" edge="1" parent="1" source="Eu_Z3l2dlq1L3zP6mrAY-10" target="Eu_Z3l2dlq1L3zP6mrAY-7">
+          <mxGeometry relative="1" as="geometry">
+            <mxPoint x="250" y="235" as="sourcePoint" />
+            <mxPoint x="554" y="200" as="targetPoint" />
+          </mxGeometry>
+        </mxCell>
+        <mxCell id="TMhfgzIf1tkrQ-7BzbZ0-3" value="KUBECONFIG" style="shape=note;whiteSpace=wrap;html=1;backgroundOutline=1;darkOpacity=0.05;strokeColor=#999999;dashed=1;fontColor=#999999;" vertex="1" parent="1">
+          <mxGeometry x="80" y="300" width="160" height="70" as="geometry" />
+        </mxCell>
+      </root>
+    </mxGraphModel>
+  </diagram>
 </mxfile>