[MySQL] Add replica status data stream (elastic#10344)

* add replica status data stream

* update data stream

* update changelog

* Address review comment

* address review comments

* address review comment

* address review comments

packages/mysql/_dev/build/docs/README.md

@@ -1,17 +1,70 @@
 # MySQL Integration
 
-This integration periodically fetches logs and metrics from [MySQL](https://www.mysql.com/) servers.
+## Overview
+
+[MySQL](https://www.mysql.com/) is an open-source Relational Database Management System (RDBMS) that enables users to store, manage, and retrieve structured data efficiently.
+
+Use the MySQL integration to:
+
+- Collect error and slow query logs, as well as status, galera status, and replication status metrics, to provide insights into database operations, query performance and replication health.
+- Create informative visualizations to track usage trends, measure key metrics, and derive actionable business insights.
+- Set up alerts to minimize Mean Time to Detect (MTTD) and Mean Time to Resolve (MTTR) by quickly referencing relevant logs during troubleshooting.
+
+## Data streams
+
+The MySQL integration collects logs and metrics data, providing comprehensive insights into database operations and performance.
+
+Logs provide insights into the operations and events within the MySQL environment. The MySQL integration collects `error` logs helping users to track errors and warnings, understand their causes, and address database-related issues efficiently. This includes monitoring for slow-performing queries through the `slowlog` data stream, which is critical for identifying and resolving queries that negatively affect database performance. 
+
+Metrics offer statistics that reflect the performance and health of MySQL. The `status` data stream, for instance, gathers a variety of performance metrics, including connection errors, cache efficiency, and InnoDB storage engine details. The `galera_status` data stream offers a view into the health and performance of Galera Clusters, which is vital for the maintenance of distributed database systems. For replication health, the `replica_status` data stream provides metrics that shed light on the state of replication between the source and replica servers, ensuring the replication process is functioning correctly. 
+
+Data streams:
+
+- `error`: Collect error logs from the MySQL server, helping to detect and troubleshoot issues that may affect database functionality. This data stream includes information such as error messages, severities, and error codes.
+- `slowlog`: Collect slow-performing queries that exceed a defined time threshold. This data stream includes details such as query execution time, lock time, rows affected, and the actual query text, which are crucial for pinpointing and optimizing slow queries.
+- `status`: Collect various status and performance indicators, including connection errors, cache performance, binary log usage, network I/O, thread activity, and detailed InnoDB metrics, allowing for a thorough analysis of the MySQL server's health and efficiency.
+- `galera_status`: Collect various status and performance metrics, which provide insights into cluster performance, including replication health and node status, to maintain the robustness and fault tolerance of the distributed database system.
+- `replica_status`:  Collect metrics related to status and performance of the replication process, including details from source and replica servers.
+
+## Note:
+- Users can monitor MySQL logs by using the logs-* index pattern in the Discover feature, while metrics can be viewed using the metrics-* index pattern.
 
 ## Compatibility
 
-The `error` and `slowlog` datasets were tested with logs from MySQL 5.5, 5.7 and 8.0, MariaDB 10.1, 10.2 and 10.3, and Percona 5.7 and 8.0.
+- Databases version compatibility across data streams.
+|Data Stream      | MySQL Version   | MariaDB Version    |Percona Version | 
+| ----------------|-----------------|--------------------|----------------|
+|error and slowlog|`5.5`,`5.7`,`8.0`|`10.1`,`10.2`,`10.3`|`5.7`,`5.8`     |
+|galera_status and status|`5.7`,`8.0`|`10.2`,`10.3`,`10.4`|`5.7`,`8.0`    |
+|replica_status|`5.7`,`8.0.22`|`10.4`,`10.5.1`|`5.7`,`8.0.22`|
+         
+## Note:
+- MySQL and Percona from version `8.0.22` onwards and MariaDB from version `10.5.1` onwards support the `SHOW REPLICA STATUS;` query. Versions prior to these use the `SHOW SLAVE STATUS;` query.
+
+## Prerequisites
+
+Users require Elasticsearch for storing and searching their data, and Kibana for visualizing and managing it. They can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on their own hardware.
+
+In order to ingest data from MySQL:
+
+- Users should specify the hostname, username, and password to connect to the MySQL database. Additionally, there is query parameter in replica_status data stream(default query is `SHOW REPLICA STATUS;` user can change it to `SHOW SLAVE STATUS`).
+- Users should specify the paths of MySQL error logs and slow logs. (default paths are:- Error logs: `/var/log/mysql/error.log*` and `/var/log/mysqld.log*`, Slow logs: `/var/log/mysql/*-slow.log*` and `/var/lib/mysql/*-slow.log*`)
 
-The `galera_status` and `status` datasets were tested with MySQL and Percona 5.7 and 8.0 and are expected to work with all
-versions >= 5.7.0. It is also tested with MariaDB 10.2, 10.3 and 10.4.
+## Setup
 
-## Logs
+For step-by-step instructions on how to set up an integration, see the [Getting started](https://www.elastic.co/guide/en/welcome-to-elastic/current/getting-started-observability.html) guide.
 
-### error
+## Validation
+
+After the integration is successfully configured, clicking on the Assets tab of the MySQL Integration should display a list of available dashboards. Click on the dashboard available for the user's configured data stream. It should be populated with the required data.
+
+## Troubleshooting
+
+For MySQL, MariaDB and Percona the query to check replica status varies depending on the version of the database. Users should adjust the query in the integration configuration accordingly. 
+
+## Logs reference
+
+### Error
 
 The `error` dataset collects the MySQL error logs.
 
@@ -21,7 +74,7 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur
 
 {{fields "error"}}
 
-### slowlog
+### Slow Log
 
 The `slowlog` dataset collects the MySQL slow logs.
 
@@ -31,9 +84,9 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur
 
 {{fields "slowlog"}}
 
-## Metrics
+## Metrics reference
 
-### galera_status
+### Galera Status
 
 The `galera_status` dataset periodically fetches metrics from [Galera](http://galeracluster.com/)-MySQL cluster servers.
 
@@ -45,7 +98,19 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur
 
 {{fields "galera_status"}}
 
-### status
+### Replica Status
+
+The `replica_status` dataset collects data from MySQL by running a `SHOW REPLICA STATUS;` or `SHOW SLAVE STATUS;` query. This data stream provides information about the configuration and status of the connection between the replica server and the source server.
+
+{{event "replica_status"}}
+
+**ECS Field Reference**
+
+Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields.
+
+{{fields "replica_status"}}
+
+### Status
 
 The MySQL `status` dataset collects data from MySQL by running a `SHOW GLOBAL STATUS;` SQL query. This query returns a large number of metrics.
 

packages/mysql/_dev/deploy/docker/Dockerfile

@@ -9,4 +9,7 @@ HEALTHCHECK --interval=1s --retries=90 CMD /healthcheck.sh
 COPY healthcheck.sh /healthcheck.sh
 COPY entrypoint.sh /test-entrypoint.sh
 
+RUN chmod +x /healthcheck.sh
+RUN chmod +x /test-entrypoint.sh
+
 ENTRYPOINT ["bash", "/test-entrypoint.sh"]

packages/mysql/_dev/deploy/docker/docker-compose.yml

@@ -9,13 +9,32 @@ services:
       - 3306
     volumes:
       - ${SERVICE_LOGS_DIR}/mysql:/var/log/mysql
-      - mysqldata:/var/lib/mysql
     environment:
       - IMAGE=${IMAGE:-mysql:8.0.35}
   mysql_is_ready:
     image: tianon/true
     depends_on:
-      mysql:
-        condition: service_healthy
-volumes:
-  mysqldata:
+      - mysql
+  mysql_master:
+    build: ./master
+    env_file:
+      - ./master/mysql_master.env
+    environment:
+      - MYSQL_ROOT_PASSWORD=111
+    volumes:
+      - ./master/conf/mysql.conf.cnf:/etc/mysql/conf.d/mysql.conf.cnf
+    ports:
+      - 3306
+  mysql_replica:
+    build: ./replica
+    env_file:
+      - ./replica/mysql_replica.env
+    environment:
+      - MYSQL_MASTER_HOST=mysql_master
+      - MYSQL_ROOT_PASSWORD=111
+    volumes:
+      - ./replica/conf/mysql.conf.cnf:/etc/mysql/conf.d/mysql.conf.cnf
+    ports:
+      - 3306
+    depends_on:
+      - mysql_master

packages/mysql/_dev/deploy/docker/master/Dockerfile

@@ -0,0 +1,6 @@
+FROM mysql:8.0.35
+ENV MYSQL_ROOT_PASSWORD=111
+COPY entrypoint-master.sh /entrypoint-master.sh
+RUN chmod +x /entrypoint-master.sh
+ENTRYPOINT ["/entrypoint-master.sh"]
+CMD ["mysqld"]

packages/mysql/_dev/deploy/docker/master/conf/mysql.conf.cnf

@@ -0,0 +1,14 @@
+[mysqld]
+
+skip-name-resolve
+default_authentication_plugin = mysql_native_password
+
+server-id = 1
+log_bin = mysql-bin
+binlog_format = ROW
+binlog_do_db = mysql
+
+gtid_mode = ON
+enforce_gtid_consistency = ON
+log_slave_updates = ON
+tls_version = TLSv1.2,TLSv1.3

packages/mysql/_dev/deploy/docker/master/entrypoint-master.sh

@@ -0,0 +1,30 @@
+#!/bin/bash
+set -e
+
+echo "Starting MySQL master..."
+
+# Initialize and start MySQL server using the default entrypoint script
+docker-entrypoint.sh mysqld &
+
+# Wait for the master database to be ready
+echo "Waiting for MySQL to be ready..."
+while ! mysqladmin ping -h "127.0.0.1" --silent; do
+    echo "MySQL master is not up yet..."
+    sleep 2
+done
+echo "MySQL master is ready."
+
+# Create replication user on the master
+# Check if the replication user already exists
+user_exists=$(mysql -uroot -p"$MYSQL_ROOT_PASSWORD" -e "SELECT 1 FROM mysql.user WHERE user = 'mydb_replica_user';" -ss)
+if [ -z "$user_exists" ]; then
+  echo "Creating replication user..."
+priv_stmt='CREATE USER "mydb_replica_user"@"%" IDENTIFIED BY "mydb_replica_pwd"; GRANT REPLICATION SLAVE ON *.* TO "mydb_replica_user"@"%"; FLUSH PRIVILEGES;'
+# docker-compose exec mysql_master mysql -uroot -p"$MYSQL_ROOT_PASSWORD" -e "$priv_stmt"
+mysql -uroot -p"$MYSQL_ROOT_PASSWORD" -e "$priv_stmt"
+else
+  echo "Replication user already exists. Skipping user creation."
+fi
+
+# Now, keep the script running to prevent the container from exiting
+wait

packages/mysql/_dev/deploy/docker/master/healthcheck.sh

@@ -0,0 +1,8 @@
+#!/bin/bash
+# Health check script for MySQL
+
+# Use the MYSQL_ROOT_PASSWORD environment variable or a default password
+MYSQL_PWD=${MYSQL_ROOT_PASSWORD:-defaultpassword}
+
+# Check if MySQL is ready
+mysqladmin ping -h localhost -u root --silent

packages/mysql/_dev/deploy/docker/master/mysql_master.env

@@ -0,0 +1,5 @@
+# Note, by default mysql root does not have a password. You need to restart a server to bring MYSQL_ROOT_PASSWORD working. Use "docker-compose restart" command.
+MYSQL_ROOT_PASSWORD=111
+MYSQL_PORT=3306
+MYSQL_USER=mydb_user
+MYSQL_PASSWORD=mydb_pwd

packages/mysql/_dev/deploy/docker/replica/Dockerfile

@@ -0,0 +1,9 @@
+FROM mysql:8.0.35
+ENV MYSQL_ROOT_PASSWORD=111
+COPY entrypoint-replica.sh /entrypoint-replica.sh
+COPY grant-permissions.sql /docker-entrypoint-initdb.d/
+COPY healthcheck.sh /healthcheck.sh
+RUN chmod +x /healthcheck.sh
+RUN chmod +x /entrypoint-replica.sh
+ENTRYPOINT ["/entrypoint-replica.sh"]
+CMD ["mysqld"]

packages/mysql/_dev/deploy/docker/replica/conf/mysql.conf.cnf

@@ -0,0 +1,13 @@
+[mysqld]
+
+skip-name-resolve
+default_authentication_plugin = mysql_native_password
+
+server-id = 2
+log_bin = mysql-bin
+binlog_do_db = mysql
+
+gtid_mode = ON
+enforce_gtid_consistency = ON
+log_slave_updates = ON
+tls_version = TLSv1.2,TLSv1.3

packages/mysql/_dev/deploy/docker/replica/entrypoint-replica.sh

@@ -0,0 +1,63 @@
+#!/bin/bash
+set -e
+
+echo "Starting MySQL slave..."
+
+# Start the MySQL server in the background
+docker-entrypoint.sh mysqld &
+
+# Wait for the slave database to be ready
+echo "Waiting for MySQL to be ready..."
+while ! mysqladmin ping -h "127.0.0.1" --silent; do
+    echo "MySQL slave is not up yet..."
+    sleep 2
+done
+echo "MySQL slave is ready."
+
+# Configure the slave to start replication
+echo "Configuring slave to start replication..."
+# Wait for the master to be ready and for its logs to be available
+MASTER_LOGS_AVAILABLE=false
+for i in {1..30}; do
+  MS_STATUS=$(mysql -h "$MYSQL_MASTER_HOST" -P 3306 -u root -p"$MYSQL_ROOT_PASSWORD" -e "SHOW MASTER STATUS;")
+  CURRENT_LOG=$(echo "$MS_STATUS" | awk 'NR==2 {print $1}')
+  CURRENT_POS=$(echo "$MS_STATUS" | awk 'NR==2 {print $2}')
+  if [ -n "$CURRENT_LOG" ] && [ -n "$CURRENT_POS" ]; then
+    MASTER_LOGS_AVAILABLE=true
+    break
+  else
+    echo "Waiting for master logs to be available..."
+    sleep 2
+  fi
+done
+
+if [ "$MASTER_LOGS_AVAILABLE" = false ]; then
+    echo "Failed to obtain master log file and position."
+    exit 1
+fi
+
+echo "Master status obtained. Current log: $CURRENT_LOG, Current position: $CURRENT_POS."
+
+# Reset the slave to ensure a clean replication setup
+echo "Resetting the slave..."
+mysql -uroot -p"$MYSQL_ROOT_PASSWORD" -e "STOP SLAVE; RESET SLAVE ALL;"
+
+# Prepare the CHANGE MASTER TO command
+start_slave_stmt="CHANGE MASTER TO MASTER_HOST='$MYSQL_MASTER_HOST', MASTER_USER='mydb_replica_user', MASTER_PASSWORD='mydb_replica_pwd', MASTER_LOG_FILE='$CURRENT_LOG', MASTER_LOG_POS=$CURRENT_POS; START SLAVE;"
+
+# Display the command that will be executed
+echo "Running the following command to start replication: $start_slave_stmt"
+
+# Run the CHANGE MASTER TO command
+mysql -uroot -p"$MYSQL_ROOT_PASSWORD" -e "$start_slave_stmt" || {
+    echo "Failed to start replication. Entering sleep mode for debugging."
+    tail -f /dev/null # This will keep the container running indefinitely
+}
+
+# Verify slave status
+echo "Verifying slave status..."
+mysql -uroot -p"$MYSQL_ROOT_PASSWORD" -e "GRANT REPLICATION CLIENT ON *.* TO 'mydb_replica_user'@'%'; FLUSH PRIVILEGES;"
+mysql -uroot -p"$MYSQL_ROOT_PASSWORD" -e "SHOW SLAVE STATUS \G"
+
+# Now, keep the script running to prevent the container from exiting
+wait

packages/mysql/_dev/deploy/docker/replica/grant-permissions.sql

@@ -0,0 +1,2 @@
+GRANT REPLICATION CLIENT ON *.* TO 'mydb_replica_user'@'%';
+FLUSH PRIVILEGES;

packages/mysql/_dev/deploy/docker/replica/healthcheck.sh

@@ -0,0 +1,8 @@
+#!/bin/bash
+# Health check script for MySQL
+
+# Use the MYSQL_ROOT_PASSWORD environment variable or a default password
+MYSQL_PWD=${MYSQL_ROOT_PASSWORD:-defaultpassword}
+
+# Check if MySQL is ready
+mysqladmin ping -h localhost -u root --silent

packages/mysql/_dev/deploy/docker/replica/mysql_replica.env

@@ -0,0 +1,5 @@
+# Note, by default mysql root does not have a password. You need to restart a server to bring MYSQL_ROOT_PASSWORD working. Use "docker-compose restart" command.
+MYSQL_ROOT_PASSWORD=111
+MYSQL_PORT=3306
+MYSQL_USER=mydb_replica_user
+MYSQL_PASSWORD=mydb_replica_pwd

packages/mysql/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: 1.24.0
+  changes:
+    - description: Add replica_status datastream.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/1
 - version: 1.23.0
   changes:
     - description: Add processor support for galera_status, performance and staus data streams.

packages/mysql/data_stream/replica_status/_dev/test/pipeline/test-common-config.yml

@@ -0,0 +1,2 @@
+dynamic_fields:
+  "event.ingested": ".*"