Merge pull request #27 from DataKitchen/release/3.0.0

Release/3.0.0

Author: datakitchen-devops

.dockerignore

@@ -15,3 +15,5 @@ node_modules/
 scripts/
 venv/
 .ruff_cache/
+deploy
+!deploy/install_*.sh

.github/workflows/python-job.yml

@@ -5,7 +5,7 @@ on:
     inputs:
       python-version:
         type: string
-        default: "3.10"
+        default: "3.12"
       run:
         required: true
         type: string

CONTRIBUTORS.md

@@ -16,3 +16,4 @@ Thanks to everyone who has contributed to DataOps TestGen!
 - [Eric Estabrooks](https://www.linkedin.com/in/ericestabrooks/) 🤔
 - [Gil Benghiat](https://www.linkedin.com/in/gilbertbenghiat/) 🤔 💬
 - [Chris Bergh](https://www.linkedin.com/in/chrisbergh/) 🤔 📖
+- [Astor Fernandez](https://www.linkedin.com/in/astorfernandez/) 💻

Dockerfile

@@ -83,14 +83,14 @@ As an alternative to the Docker Compose [installation with dk-installer (recomme
 
 | Software                                                                                                                                                                         | Tested Versions  | Command to check version               |
 |----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------|------------------------------|
-| [Python](https://www.python.org/downloads/) <br/>- Most Linux and macOS systems have Python pre-installed. <br/>- On Windows machines, you will need to download and install it. | 3.10, 3.11, 3.12 | `python3 --version`               |
+| [Python](https://www.python.org/downloads/) <br/>- Most Linux and macOS systems have Python pre-installed. <br/>- On Windows machines, you will need to download and install it. | 3.12 | `python3 --version`               |
 | [PostgreSQL](https://www.postgresql.org/download/)                                                                                                                                                                     | 14.1, 15.8, 16.4       | `psql --version`|
 
 ### Install the TestGen package
 
 We recommend using a Python virtual environment to avoid any dependency conflicts with other applications installed on your machine. The [venv](https://docs.python.org/3/library/venv.html#creating-virtual-environments) module, which is part of the Python standard library, or other third-party tools, like [virtualenv](https://virtualenv.pypa.io/en/latest/) or [conda](https://docs.conda.io/en/latest/), can be used.
 
-Create and activate a virtual environment with a TestGen-compatible version of Python (`>=3.10`). The steps may vary based on your operating system and Python installation - the [Python packaging user guide](https://packaging.python.org/en/latest/tutorials/installing-packages/) is a useful reference.
+Create and activate a virtual environment with a TestGen-compatible version of Python (`>=3.12`). The steps may vary based on your operating system and Python installation - the [Python packaging user guide](https://packaging.python.org/en/latest/tutorials/installing-packages/) is a useful reference.
 
 _On Linux/Mac_
 ```shell
@@ -100,7 +100,7 @@ source venv/bin/activate
 
 _On Windows_
 ```powershell
-py -3.10 -m venv venv
+py -3.12 -m venv venv
 venv\Scripts\activate
 ```
 
@@ -144,7 +144,7 @@ Source the file to apply the environment variables. For the Windows equivalent,
 source local.env
 ```
 
-Make sure the PostgreSQL database server is up and running. Initialize the application database for TestGen. 
+Make sure the PostgreSQL database server is up and running. Initialize the application database for TestGen.
 ```shell
 testgen setup-system-db --yes
 ```

README.md

@@ -1,23 +1,32 @@
+variable "TESTGEN_LABELS" {}
+variable "TESTGEN_BASE_LABEL" {}
 variable "TESTGEN_VERSION" {}
-variable "PUBLIC_RELEASE" {
-  default = false
-}
 
-function "docker_repo" {
-  params = []
-  result = PUBLIC_RELEASE ? "datakitchen/dataops-testgen" : "datakitchen/dataops-testgen-qa"
+target "testgen-release" {
+  args = {
+    TESTGEN_VERSION = "${TESTGEN_VERSION}"
+    TESTGEN_BASE_LABEL = "${TESTGEN_BASE_LABEL}"
+  }
+  context = "."
+  dockerfile = "deploy/testgen.dockerfile"
+  platforms = ["linux/amd64", "linux/arm64"]
+  tags = formatlist("datakitchen/dataops-testgen:%s", split(" ", TESTGEN_LABELS))
 }
 
-target "testgen" {
+target "testgen-qa" {
   args = {
     TESTGEN_VERSION = "${TESTGEN_VERSION}"
+    TESTGEN_BASE_LABEL = "${TESTGEN_BASE_LABEL}"
   }
   context = "."
-  dockerfile = "Dockerfile"
+  dockerfile = "deploy/testgen.dockerfile"
+  platforms = ["linux/amd64", "linux/arm64"]
+  tags = [format("datakitchen/dataops-testgen-qa:%s", index(split(" ", TESTGEN_LABELS), 0))]
+}
+
+target "testgen-base" {
+  context = "."
+  dockerfile = "deploy/testgen-base.dockerfile"
   platforms = ["linux/amd64", "linux/arm64"]
-  tags = [
-    "${docker_repo()}:v${TESTGEN_VERSION}",
-    PUBLIC_RELEASE ? "${docker_repo()}:v${index(regex("([0-9]+.[0-9]+).[0-9]+", TESTGEN_VERSION), 0)}": "",
-    PUBLIC_RELEASE ? "${docker_repo()}:v${index(regex("([0-9]+).[0-9]+.[0-9]+", TESTGEN_VERSION), 0)}": ""
-  ]
+  tags = formatlist("datakitchen/dataops-testgen-base:%s", split(" ", TESTGEN_LABELS))
 }

deploy/docker-bake.hcl

@@ -0,0 +1,50 @@
+#!/usr/bin/env bash
+
+export ARROW_VERSION=18.0.0
+export ARROW_SHA256=9c473f2c9914c59ab571761c9497cf0e5cfd3ea335f7782ccc6121f5cb99ae9b
+
+export ARROW_HOME=/dk
+export PARQUET_HOME=/dk
+
+mkdir /arrow
+
+# Obtaining and expanding Arrow
+wget -q https://github.com/apache/arrow/archive/apache-arrow-${ARROW_VERSION}.tar.gz -O /tmp/apache-arrow.tar.gz
+echo "${ARROW_SHA256} *apache-arrow.tar.gz" | sha256sum /tmp/apache-arrow.tar.gz
+tar -xvf /tmp/apache-arrow.tar.gz -C /arrow --strip-components 1
+
+pushd /arrow/cpp
+
+# Configure the build using CMake
+cmake --preset ninja-release-python
+
+# Configuring cmake for ARM only
+if [ "$(uname -m)" = "arm64" ] || [ "$(uname -m)" = "aarch64" ]; then
+  cmake -DCMAKE_CXX_FLAGS="-march=armv8-a"
+fi
+
+# Pre-fetch dependencies without building
+cmake --build . --target re2_ep -- -j2 || true
+
+# Apply the patch to re2 after the dependencies are fetched but before the build
+pushd re2_ep-prefix/src/re2_ep
+
+cat <<EOF | patch -p1
+diff --git a/util/pcre.h b/util/pcre.h
+--- a/util/pcre.h
++++ b/util/pcre.h
+@@ -166,0 +166,1 @@
++#include <cstdint>
+EOF
+
+popd
+
+# Finish processing dependencies after patch
+cmake --build . --target re2_ep -- -j2
+
+# Continue with the build and install Apache Arrow
+cmake --build . --target install
+
+popd
+
+rm -rf /arrow /tmp/apache-arrow.tar.gz

deploy/install_arrow.sh

@@ -0,0 +1,32 @@
+#!/usr/bin/env bash
+
+# From: https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server
+# modifications: Added --non-interactive and --no-cache flags, removed sudo, added aarch64 as an alias for arm64
+
+case $(uname -m) in
+    x86_64)   architecture="amd64" ;;
+    arm64)   architecture="arm64" ;;
+    aarch64)   architecture="arm64" ;;
+    *) architecture="unsupported" ;;
+esac
+if [[ "unsupported" == "$architecture" ]];
+then
+    echo "Alpine architecture $(uname -m) is not currently supported.";
+    exit;
+fi
+
+#Download the desired package(s)
+curl -O https://download.microsoft.com/download/7/6/d/76de322a-d860-4894-9945-f0cc5d6a45f8/msodbcsql18_18.4.1.1-1_$architecture.apk
+curl -O https://download.microsoft.com/download/7/6/d/76de322a-d860-4894-9945-f0cc5d6a45f8/mssql-tools18_18.4.1.1-1_$architecture.apk
+
+#(Optional) Verify signature, if 'gpg' is missing install it using 'apk add gnupg':
+curl -O https://download.microsoft.com/download/7/6/d/76de322a-d860-4894-9945-f0cc5d6a45f8/msodbcsql18_18.4.1.1-1_$architecture.sig
+curl -O https://download.microsoft.com/download/7/6/d/76de322a-d860-4894-9945-f0cc5d6a45f8/mssql-tools18_18.4.1.1-1_$architecture.sig
+
+curl https://packages.microsoft.com/keys/microsoft.asc  | gpg --import -
+gpg --verify msodbcsql18_18.4.1.1-1_$architecture.sig msodbcsql18_18.4.1.1-1_$architecture.apk
+gpg --verify mssql-tools18_18.4.1.1-1_$architecture.sig mssql-tools18_18.4.1.1-1_$architecture.apk
+
+#Install the package(s)
+apk add --no-cache --non-interactive --allow-untrusted msodbcsql18_18.4.1.1-1_$architecture.apk
+apk add --no-cache --non-interactive --allow-untrusted mssql-tools18_18.4.1.1-1_$architecture.apk

deploy/install_linuxodbc.sh

@@ -0,0 +1,82 @@
+FROM python:3.12.7-alpine3.20
+
+ENV LANG=C.UTF-8
+ENV LC_ALL=C.UTF-8
+ENV PYTHONDONTWRITEBYTECODE=1
+ENV PYTHONFAULTHANDLER=1
+ENV ACCEPT_EULA=Y
+
+RUN apk update && apk add --no-cache \
+    gcc \
+    g++ \
+    bash \
+    libffi-dev \
+    openssl-dev \
+    cargo \
+    musl-dev \
+    postgresql-dev \
+    cmake \
+    rust \
+    linux-headers \
+    libc-dev \
+    libgcc \
+    libstdc++ \
+    ca-certificates \
+    zlib-dev \
+    bzip2-dev \
+    xz-dev \
+    lz4-dev \
+    zstd-dev \
+    snappy-dev \
+    brotli-dev \
+    build-base \
+    autoconf \
+    boost-dev \
+    flex \
+    libxml2-dev \
+    libxslt-dev \
+    libjpeg-turbo-dev \
+    ninja \
+    git \
+    curl \
+    unixodbc-dev \
+    gpg \
+    openssl=3.3.2-r1 \
+    gfortran \
+    openblas-dev
+
+RUN mkdir /dk
+
+COPY --chmod=775 ./deploy/install_linuxodbc.sh /tmp/dk/install_linuxodbc.sh
+RUN /tmp/dk/install_linuxodbc.sh
+
+COPY --chmod=775 ./deploy/install_arrow.sh /tmp/dk/install_arrow.sh
+RUN /tmp/dk/install_arrow.sh
+
+# Install TestGen's main project empty pyproject.toml to install (and cache) the dependencies first
+COPY ./pyproject.toml /tmp/dk/pyproject.toml
+RUN python3 -m pip install --prefix=/dk /tmp/dk
+
+RUN apk del \
+    gcc \
+    g++ \
+    bash \
+    libffi-dev \
+    openssl-dev \
+    cargo \
+    musl-dev \
+    postgresql-dev \
+    cmake \
+    rust \
+    linux-headers \
+    libc-dev \
+    build-base \
+    autoconf \
+    boost-dev \
+    flex \
+    ninja \
+    curl \
+    unixodbc-dev \
+    gpg \
+    ca-certificates \
+    git

deploy/testgen-base.dockerfile

@@ -0,0 +1,38 @@
+ARG TESTGEN_BASE_LABEL
+ARG TESTGEN_VERSION
+
+FROM datakitchen/dataops-testgen-base:${TESTGEN_BASE_LABEL} as build-image
+
+# Now install everything
+COPY . /tmp/dk/
+RUN python3 -m pip install --prefix=/dk /tmp/dk
+
+FROM python:3.12.7-alpine3.20 AS release-image
+
+RUN addgroup -S testgen && adduser -S testgen -G testgen
+
+COPY --from=build-image --chown=testgen:testgen /dk/ /dk
+COPY --from=build-image /usr/local/lib/ /usr/local/lib
+COPY --from=build-image /usr/lib/ /usr/lib
+COPY --from=build-image /opt/microsoft/ /opt/microsoft
+COPY --from=build-image /etc/odbcinst.ini /etc/odbcinst.ini
+
+# The OpenSSL upgrade is not carried from the build image, so we have to upgrade it again
+#RUN apk add --no-cache openssl=3.3.2-r1
+
+ENV PYTHONPATH=/dk/lib/python3.12/site-packages
+ENV PATH="$PATH:/dk/bin:/opt/mssql-tools/bin/"
+
+ENV TESTGEN_VERSION=${TESTGEN_VERSION}
+ENV TG_RELEASE_CHECK=docker
+ENV TESTGEN_DOCKER_HUB_REPO=datakitchen/dataops-testgen-enterprise
+ENV STREAMLIT_SERVER_MAX_UPLOAD_SIZE=200
+
+RUN mkdir /var/lib/testgen && chown testgen:testgen /var/lib/testgen
+
+USER testgen
+
+WORKDIR /dk
+
+ENTRYPOINT ["testgen"]
+CMD [ "ui", "run" ]

deploy/testgen.dockerfile

@@ -10,7 +10,7 @@ This document describes how to set up your local environment for TestGen develop
 
 ### Clone repository
 
-Login to your GitHub account. 
+Login to your GitHub account.
 
 Fork the [dataops-testgen](https://github.com/DataKitchen/dataops-testgen) repository, following [GitHub's guide](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/fork-a-repo).
 
@@ -23,17 +23,17 @@ git clone https://github.com/YOUR-USERNAME/dataops-testgen
 
 We recommend using a Python virtual environment to avoid any dependency conflicts with other applications installed on your machine. The [venv](https://docs.python.org/3/library/venv.html#creating-virtual-environments) module, which is part of the Python standard library, or other third-party tools, like [virtualenv](https://virtualenv.pypa.io/en/latest/) or [conda](https://docs.conda.io/en/latest/), can be used.
 
-From the root of your local repository, create and activate a virtual environment with a TestGen-compatible version of Python (`>=3.10`). The steps may vary based on your operating system and Python installation - the [Python packaging user guide](https://packaging.python.org/en/latest/tutorials/installing-packages/) is a useful reference.
+From the root of your local repository, create and activate a virtual environment with a TestGen-compatible version of Python (`>=3.12`). The steps may vary based on your operating system and Python installation - the [Python packaging user guide](https://packaging.python.org/en/latest/tutorials/installing-packages/) is a useful reference.
 
 _On Linux/Mac_
 ```shell
-python3.10 -m venv venv
+python3.12 -m venv venv
 source venv/bin/activate
 ```
 
 _On Windows_
 ```powershell
-py -3.10 -m venv venv
+py -3.12 -m venv venv
 venv\Scripts\activate
 ```
 
@@ -84,7 +84,7 @@ Run a PostgreSQL instance as a Docker container.
 docker compose -f docker-compose.local.yml up -d
 ```
 
-Initialize the application database for TestGen. 
+Initialize the application database for TestGen.
 ```shell
 testgen setup-system-db --yes
 ```