Add READMEs

Author: mostafa

README.md

@@ -1,112 +1,91 @@
 # DeepSQLi
 
-This repository contains the code for the DeepSQLi project. The project aims to detect SQL injection attacks using deep learning models. The project consists of two main components: **Tokenizer API** and **Serving API**. The Tokenizer API tokenizes and sequences the input query, while the Serving API predicts whether the query is SQL injection or not.
+**DeepSQLi** is a project designed to detect SQL injection (SQLi) attacks using deep learning models. It provides a **Prediction API**, a Flask application that tokenizes and sequences SQL queries, and then predicts whether a query is an SQL injection.
 
-The Tokenizer API is built using Flask and TensorFlow, and the Serving API is built using TensorFlow Serving. The Tokenizer API is responsible for tokenizing and sequencing the input query using the corresponding [dataset](./dataset/), and the Serving API is responsible for predicting whether the query is SQL injection or not using the trained deep learning [model](./sqli_model/).
+## Project Overview
 
-The project also includes a [SQL IDS/IPS plugin](https://github.com/gatewayd-io/gatewayd-plugin-sql-ids-ips) that integrates with the GatewayD database gatewayd. The plugin serves as a frontend for these APIs. It intercepts the incoming queries and sends them to the Serving API for prediction. If the query is predicted as SQL injection, the plugin terminates the request; otherwise, it forwards the query to the database.
+The **Prediction API** handles both tokenization and prediction in a single endpoint, simplifying the detection workflow. It processes incoming SQL queries and determines if they contain SQL injection patterns.
 
-The following diagram shows the architecture of the project:
+### GatewayD Integration
+
+This project can be integrated with [GatewayD](https://github.com/gatewayd-io/gatewayd) using the [GatewayD SQL IDS/IPS plugin](https://github.com/gatewayd-io/gatewayd-plugin-sql-ids-ips). The plugin acts as a middleware between clients and the database, intercepting SQL queries and sending them to the Prediction API for analysis. If a query is classified as malicious by the Prediction API, the plugin blocks the query; otherwise, it forwards the query to the database.
+
+### Architecture
 
 ```mermaid
 flowchart TD
     Client <-- PostgreSQL wire protocol:15432 --> GatewayD
-    GatewayD <--> Sq["SQL IDS/IPS plugin"]
-    Sq <-- http:8000 --> T["Tokenizer API"]
-    Sq <-- http:8501 --> S["Serving API"]
-    S -- loads --> SM["SQLi models"]
-    T -- loads --> Dataset
-    Sq -- threshold: 80% --> D{Malicious query?}
+    GatewayD <--> P["Prediction API"]
+    P --loads--> SM["SQLi Model"]
+    P --threshold: 80% --> D{Malicious query?}
     D -->|No: send to| Database
     D -->|Yes: terminate request| GatewayD
 ```
 
-There are currently two models available and trained using the [dataset](./dataset/). Both models are trained using the same model architecture, but they are trained using different datasets. The first model is trained using the [SQLi dataset v1](./dataset/sqli_dataset1.csv), and the second model is trained using the [SQLi dataset v2](./dataset/sqli_dataset2.csv). The models are trained using the following hyperparameters:
-
-- Model architecture: LSTM (Long Short-Term Memory)
-- Embedding dimension: 128
-- LSTM units: 64
-- Dropout rate: 0.2
-- Learning rate: 0.001
-- Loss function: Binary crossentropy
-- Optimizer: Adam
-- Metrics: Accuracy, precision, recall, and F1 score
-- Validation split: 0.2
-- Dense layer units: 1
-- Activation function: Sigmoid
-- Maximum sequence length: 100
-- Maximum number of tokens: 10000
-- Maximum number of epochs: 11
-- Batch size: 32
+## Models and Tokenization
 
-## Installation
+### Model Versions
 
-The fastest way to get started is to use Docker and Docker Compose. If you don't have Docker installed, you can install it by following the instructions [here](https://docs.docker.com/get-docker/).
+- **LSTM Models**: The first two models are LSTM-based and are trained using different datasets:
+  - **`sqli_model/1`**: Trained on dataset v1 (for historical purposes, will remove in the future).
+  - **`sqli_model/2`**: Trained on dataset v2 (for historical purposes, will remove in the future).
 
-### Docker Compose
+- **CNN-LSTM Model**: The third model, **`sqli_model/3`**, is a hybrid CNN-LSTM model with a custom SQL tokenizer to improve performance on SQL injection patterns (recommended).
 
-Use the following command to build and run the Tokenizer and Serving API containers using Docker Compose (recommended). Note that `--build` is only needed the first time, and you can omit it later.
+### Tokenization
 
-```bash
-docker compose up --build -d
-```
+The Prediction API performs tokenization internally:
+
+- **Default Tokenizer** (Models 1 and 2): Uses Keras’s `Tokenizer` for general tokenization.
+- **Custom SQL Tokenizer** (Model 3): A custom tokenizer designed to handle SQL syntax and injection-specific patterns.
 
-To stop the containers, use the following command:
+## Installation
+
+### Docker Compose
+
+To start the Prediction API with Docker Compose:
 
 ```bash
-docker compose stop
+docker compose up --build -d
 ```
 
-To remove the containers and release their resources, use the following command:
+To stop and remove the containers:
 
 ```bash
 docker compose down
 ```
 
-### Docker
+### Docker (Manual Setup)
 
-#### Build the images
+#### Build Images
 
 ```bash
-docker build --no-cache --tag tokenizer-api:latest -f Dockerfile.tokenizer-api .
-docker build --no-cache --tag serving-api:latest -f Dockerfile.serving-api .
+docker build --no-cache --tag prediction-api:latest -f Dockerfile .
 ```
 
-#### Run the containers
+#### Run the Container
 
 ```bash
-docker run --rm --name tokenizer-api -p 8000:8000 -d tokenizer-api:latest
-docker run --rm --name serving-api -p 8500-8501:8500-8501 -d serving-api:latest
+docker run --rm --name prediction-api -p 8000:8000 -d prediction-api:latest
 ```
 
-### Test
-
-You can test the APIs using the following commands:
+## Usage
 
-#### Tokenizer API
-
-```bash
-# Tokenize and sequence the query
-curl 'http://localhost:8000/tokenize_and_sequence' -X POST -H 'Accept: application/json' -H 'Content-Type: application/json' --data-raw '{"query":"select * from users where id = 1 or 1=1"}'
-```
+Once the Prediction API is running, use the `predict` endpoint to classify SQL queries.
 
-#### Serving API
+### Prediction API
 
 ```bash
-# Predict whether the query is SQLi or not
-curl 'http://localhost:8501/v1/models/sqli_model/versions/3:predict' -X POST -H 'Accept: application/json' -H 'Content-Type: application/json' --data-raw '{"inputs":[[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,2,21,4,32,3,10,3,3]]}'
+curl 'http://localhost:8000/predict' -X POST -H 'Content-Type: application/json' \
+--data-raw '{"query":"SELECT * FROM users WHERE id=1 OR 1=1;"}'
 ```
 
-#### One-liner
+### Response Format
 
-```bash
-# Or you can use the following one-liner:
-curl -s 'http://localhost:8501/v1/models/sqli_model/versions/3:predict' -X POST -H 'Accept: application/json' -H 'Content-Type: application/json' --data-raw '{"inputs":['$(curl -s 'http://localhost:8000/tokenize_and_sequence' -X POST -H 'Accept: application/json' -H 'Content-Type: application/json' --data-raw '{"query":"select * from users where id = 1 or 1=1"}' | jq -c .tokens)']}' | jq
-```
+The response includes the prediction (`1` for SQL injection, `0` for legitimate query) and confidence score. Note that the confidence score is only available for the CNN-LSTM model using the Prediction API.
 
-#### Model metadata
-
-```bash
-# Get the model metadata
-curl -X GET 'http://localhost:8501/v1/models/sqli_model/versions/3/metadata'
+```json
+{
+    "confidence": 0.9722,
+}
 ```

training/README.md

@@ -0,0 +1,44 @@
+# SQL Injection Detection Model Training and Tokenization
+
+This repository contains code for training SQL injection detection models using various deep learning architectures and tokenization methods. The models are saved in the [`sqli_model`](../sqli_model/) directory, organized by versions based on their architecture and tokenization strategy.
+
+## Overview of Training Files
+
+### Model Training Scripts
+
+- **[`train.py`](train.py)**: This script trains models saved in [`sqli_model/1`](../sqli_model/1/) and [`sqli_model/2`](../sqli_model/2/). It utilizes an **LSTM-based architecture** and the default Keras tokenizer.
+- **[`train_v3.py`](train_v3.py)**: This script trains the model saved in [`sqli_model/3`](../sqli_model/3/), which uses a **Deep Learning CNN-LSTM hybrid model** with a custom SQL tokenizer designed to handle SQL syntax and injection patterns more effectively.
+
+### Tokenization Methods
+
+Each training script employs a different tokenization strategy, suitable for the specific model architecture:
+
+- **Keras Default Tokenizer (`train.py` for `sqli_model/1` and `sqli_model/2`)**:
+  - The default Keras tokenizer (`tensorflow.keras.preprocessing.text.Tokenizer`) is used in `train.py`. It performs basic tokenization by splitting the text into words and mapping each word to an integer. This simple approach is effective for general text but may miss some nuances specific to SQL syntax.
+
+- **Custom SQL Tokenizer (`train_v3.py` for `sqli_model/3`)**:
+  - The `train_v3.py` script employs a [custom SQL tokenizer](sql_tokenizer.py) specifically designed to handle SQL syntax and keywords. This tokenizer tokenizes SQL queries by recognizing SQL-specific patterns, operators, and punctuation, providing a more robust representation for SQL injection detection. This method captures complex SQL expressions that are crucial for detecting injection patterns accurately.
+
+## Model Architectures
+
+- **LSTM Model** (`train.py` for `sqli_model/1` and `sqli_model/2`):
+  - The models in `sqli_model/1` and `sqli_model/2` are trained using an LSTM (Long Short-Term Memory) network. LSTMs are particularly suited for sequential data, making them effective in capturing dependencies in SQL query patterns.
+
+- **CNN-LSTM Hybrid Model** (`train_v3.py` for sqli_model/3):
+  - The model in `sqli_model/3` is a hybrid CNN-LSTM model. It combines convolutional layers to detect local patterns in SQL syntax and LSTM layers to capture sequential dependencies. This architecture, combined with the custom SQL tokenizer, enhances the model’s ability to detect complex injection patterns.
+
+## How to Use
+
+1. **Training**:
+   - Run `train.py` to train models using the default Keras tokenizer and save them in `sqli_model/1` or `sqli_model/2`.
+   - Run `train_v3.py` to train the CNN-LSTM model with the custom SQL tokenizer and save it in `sqli_model/3`.
+
+2. **Tokenization**:
+   - The default tokenizer from Keras is used automatically within `train.py`.
+   - For `train_v3.py`, the custom SQL tokenizer defined in [sql_tokenizer.py](sql_tokenizer.py) is automatically used.
+
+## File Structure
+
+- **[`train.py`](train.py)** - Trains models with default Keras tokenizer and LSTM architecture.
+- **[`train_v3.py`](train_v3.py)** - Trains model with custom SQL tokenizer and CNN-LSTM architecture.
+- **[`sql_tokenizer.py`](sql_tokenizer.py)** - Custom SQL tokenizer for handling SQL-specific patterns, used by `train_v3.py`.

vulnerable_app/README.md

@@ -1,6 +1,17 @@
-## How to run
+# Vulnerable Customer App
+
+This is a simple, intentionally vulnerable web application designed for testing SQL injection (SQLi) detection. The app is built with Flask, connects to a PostgreSQL database via GatewayD, and exposes a vulnerability in the `/customer/<customer_id>` endpoint that allows SQL injection through unsanitized user input.
+
+> [!WARNING]
+> This application is vulnerable to SQL injection and is for testing purposes only. Do not deploy this application in production or on a public server.
+
+## Setup and Installation
 
 ```bash
+git clone git@github.com:gatewayd-io/DeepSQLi.git
+cd DeepSQLi/vulnerable_app
 pip install -r requirements.txt
 python main.py
 ```
+
+The app will start in debug mode and listen on `http://localhost:3000`.