Feat/refactor readme

Makefile

@@ -1,3 +1,5 @@
+STIX_MODEL_PATH=stix/examples/stix-attack-model.json
+
 all: install-matcher install-neo4j insert-attack-models
 
 install-matcher:
@@ -11,14 +13,10 @@ install-neo4j:
 forward-neo4j:
 	kubectl port-forward -n neo4j service/neo4j-poc 7687:7687
 
-# For now, needs patternmatcher package installed in active python virtual environment
-# and port-forwarding of neo4j using forward-neo4j rule
 insert-attack-models:
 	POD_NAME=$$(kubectl get pods -n redpanda -l app=matcher -o jsonpath='{.items[0].metadata.name}') ;\
-	kubectl cp stix/code/stix-attack-tree.json redpanda/$${POD_NAME}:/tmp/. ;\
-	kubectl exec -it -n redpanda $${POD_NAME} -- python /app/src/patternmatcher/load.py /tmp/stix-attack-tree.json
-
-
+	kubectl cp ${STIX_MODEL_PATH} redpanda/$${POD_NAME}:/tmp ;\
+	kubectl exec -it -n redpanda $${POD_NAME} -- python /app/src/patternmatcher/load.py /tmp/$(notdir ${STIX_MODEL_PATH})
 
 destroy-matcher:
 	kubectl delete pods matcher -n redpanda

README.md

@@ -1,33 +1,88 @@
-# K8sStormCenter Threatintel
-Collect tetragon logs from a honeycluster and transform them to STIX to load them into Neo4J.
+# K8sStormCenter ThreatIntel
 
-![Attack Tree](./stix/figures/stix-attack-tree.jpg)
+Welcome to the K8sStormCenter ThreatIntel repository. This repository is an integral part of the K8sStormCenter project, focusing specifically on the threat intelligence aspects and offering tools and guidance needed to analyze and identify **juicy** attacks on your honeyclusters. This repository complements the [HoneyCluster](https://github.com/k8sstormcenter/honeycluster) repository, which provides the necessary tooling for deploying your own honeyclusters and baits.
 
-## Structure
-In the neo4j directory, the code to connect to a Kafka stream of Tetragon logs, convert that stream to STIX format and write the transformed objects to Neo4J is contained.
+![Attack Tree](figures/stix-attack-tree.jpg)
 
-In the stix directory, attack trees in STIX format are collected as json files.
 
+## Purpose of This Repository
+The primary focus of this repository is to equip users with the tools and knowledge necessary for:
+- **Converting Logs**: Transforming honey cluster logs into a unified STIX format, facilitating standardized threat analysis.
+- **Building Attack Models & Indicators**: Assisting users in creating their own STIX attack models and indicators necessary for identifying pertinent attacks on their clusters.
+- **Matching Logic**: Featuring a runnable container that consumes honey cluster logs, assessing them against provided attack models to determine if they are attack-related.
 
-The conversion from Tetragon logs to STIX and uploading those logs to a Neo4J is done by [patternmatcher](./pattern_matcher), which can be built and run using the provided [Dockerfile](./pattern_matcher/Dockerfile).
+## Repository Structure
+```
+.
+├── pattern_matcher
+│   ├── Dockerfile
+│   ├── src
+│   │   └── patternmatcher
+│   └── README.md
+│
+├── stix
+│   ├── examples
+│   └── README.md
+│
+├── neo4j
+├── Makefile
+└── README.md
+```
+
+### Pattern Matcher
+The [pattern_matcher](./pattern_matcher) directory provides a Python package which is your toolkit for transforming logs from e.g. a honey cluster to STIX models and interacting with the threat database. For detailed information take a look at the pattern matcher [README](./pattern_matcher/README.md). The pattern matcher is responsible for:
+
+- **Transforming Logs**: Convert logs produced by a honeycluster into STIX observables.
+- **STIX Management**: Manage and upload STIX attack models and indicators into the threat database.
+- **Log Matching/ Threat detection**: A runnable container that consumes honey cluster logs, assessing them against provided attack models to determine if they are attack-related.
+
+### STIX Examples
+The [stix](./stix) directory provides examples and resources to help users understand how to utilize STIX data structures for their own threat intelligence and attack detection needs:
+- **STIX Attacks and Indicators**: Offers examples and guidelines on defining and customizing STIX attacks and indicators. These resources should aid users in building their own attack models tailored to detect malicious behavior  their honeyclusters.
+- **STIX Observables**: Examples how STIX observables from raw data may look like.
 
-## Getting started
-To get started a running Neo4J instance is required, which will be used to upload the STIX logs to. Either run Neo4J directly on docker using the [docker compose](./neo4j/docker-compose.yaml) or deploy it to Kind / Kubernetes following this [setup](./neo4j/README.md).
 
 
-In order to load data, the patternmatcher package needs to be installed by following this [setup](./pattern_matcher/README.md).
-Exported tetragon logs in json can be loaded using the local load [script](./pattern_matcher/scripts/load_local.sh), which can be helpful when manually analyzing a single attack.
+## Getting Started
+Follow these detailed steps to deploy the necessary components on your honey cluster, create your attack model, upload it, and set up monitoring for your specific attack interests:
+
+
+### Step 1: Deploy the Threat Database
+Begin by deploying the Neo4J database which will store and manage your threat data. This rule located in [Makefile](./Makefile) will deploy neo4j to your cluster by using helm:
+
+```bash
+make install-neo4j
+```
 
-For the automated loading, the Kafka stream of Tetragon logs has to be forwarded to the patternmatcher Docker image (by default patternmatcher consumes from port 9093).
-The patternmatcher uploads to Neo4J after transformation of the data (by default to port 7687).
-Once both ports are forwarded for the machine running patternmatcher and patternmatcher is installed, it can be run by:
+### Step 2: Deploy the Matcher Container
+Once the database is set up, deploy the pattern matcher container. It consumes a stream of the logs produced by the honeycluster, looking for signs of the specific attacks you are monitoring (based on the Indicators uploaded to Neo4J, which is done in the next step). Using the following rule [resources.yaml](./pattern_matcher/resources.yaml) is applied to your cluster:
 
 ```bash
-python -m patternmatcher.main --neo_uri <uri_to_neo4j> --kafka_uri <uri_to_kafka_stream>
+make install-matcher
 ```
+This action creates a Kubernetes pod that runs the `k8sstormcenter/matcher` container (You can have a look at the [patternmatcher](./pattern_matcher) directory to see the pattern matchers functionality and how to build your own matcher).
 
 
+### Step 3: Create and Upload Your STIX Models
+Now, focus on defining what you are detecting/defending against by creating your STIX attack models and indicators:
 
+1. **Create Your Attack Model and Indicators**: Utilize the templates and guidelines found in the `stix` directory to construct a STIX model that represents the types of attacks your system should detect.
 
+2. **Upload Your Model**:
+   - **Automated Upload**: For a quick setup, you can upload a predefined attack model [HostPath Volume Exploitation attack](./stix/examples/stix-attack-model.json) by running:
+     ```bash
+     make insert-attack-models
+     ```
+   - **Manual Upload**: If you are deploying your own attack model and indicators, upload them to the threat database using the following command:
+     ```bash
+     make insert-attack-models STIX_MODEL_PATH=path/to/stix/attack/model.json
+     ```
 
+### Step 4: Monitor and Verify Attacks
+After setting up your models, you can either simulate an attack on the honey cluster or monitor for real attacks. To verify if an attack has been detected, use the following command to port forward to the Neo4j database on the cluster and access it on your local machine:
+  ```bash
+  make forward-neo4j
+  ```
+Once connected, navigate to `localhost:7474` in your web browser to check the database entries and see if the matcher has detected any relevant attacks as defined by your models. With some luck you should be able to see lots of colorful detected balls.
 
+[![Detection](./figures/log-detection.png)](https://drive.google.com/file/d/1RfPr_7RmXDlU22-l7ZFoMnWJKloP0VpG/view?usp=sharing)

pattern_matcher/resources.yaml

@@ -18,12 +18,4 @@ spec:
         resources:
           limits:
             memory: "128Mi"
-            cpu: "500m"
-
-
-
-
-
-
-      # command: ["python", "src/patternmatcher/main.py"]
-      # args: ["--neo_uri", "bolt://neo4j-poc.neo4j.svc.cluster.local:7687", "--neo_user", "neo4j", "--neo_pass", "password", "--kafka_uri", "redpanda-src.redpanda.svc.cluster.local:9093", "--kafka_topic", "cr1"]
+            cpu: "500m"

stix/README.md

@@ -8,7 +8,7 @@ For current Tetragon traces, we therefore can make use of following SCOs: [Proce
 
 For future inclusion of Tetragon network logs, we could simply reference Network Objects in the Observed Data instead of Process Objects.
 
-![STIX2.1 Modelling of a Tetragon Log](figures/stix-log.jpg)
+![STIX2.1 Modelling of a Tetragon Log](../figures/stix-log.jpg)
 
 ### STIX Code
 
@@ -136,7 +136,7 @@ For future inclusion of Tetragon network logs, we could simply reference Network
 
 ## Attack Tree
 The following shows a STIX graph representation of the HostPath-Log-Symlink Attack Tree described in the honeycluster repo, including some Indicators that should help in identifying attacker logs. More about it below:
-![STIX2.1 Modelling of a Attack Tree](figures/stix-attack-tree.jpg)
+![STIX2.1 Modelling of a Attack Tree](../figures/stix-attack-tree.jpg)
 
 To model a theoretical attack tree and attach helpful indicators using STIX 2.1, we utilize three STIX objects: