Rollover and alias for stream indices #591

AlexRuiz7 · 2024-12-11T17:41:01Z

Description

Related issues:

One of the main requirements of the Data Persistence Model Redesign project is to include aliases and rollover policies to stream indices by default, as Index Management related features.

For Wazuh 5, we have identified 2 stream indices:

wazuh-alerts data stream.
wazuh-commands data stream.

The setup plugin (see wazuh/wazuh-indexer-plugins#9) generates indices for both data streams at startup, wazuh-alerts-5.x-0001 and .commands respectively.

On this issue, we are going to create aliases and rollover policies for both data streams, defining the rollover criteria.

We have not yet found a simple way of interacting with the OpenSearch's Indexer Management plugin, which is responsible for these things. As part of this issue, we will investigate how to implement these features within our setup plugin.

Functional requirements

The wazuh-alerts data stream is associated to an alias.
The wazuh-alerts data stream is managed by an active rollover policy.
The wazuh-commands data stream is associated to an alias.
The wazuh-commands data stream is managed by an active rollover policy.
Aliases and rollover policies are generated automatically.

Implementation restrictions

The initialization of the index aliases and the rollover policies are the responsibility of the setup plugin.

Plan

Spike. Investigate how the IM plugin persists such data.
Spike. Reproduce the IM creation of policies.
Define aliases names.
Define rollover policies.
Checkpoint
Apply changes.

The text was updated successfully, but these errors were encountered:

mcasas993 · 2025-01-16T16:11:48Z

Investigation of how the ISM plugin persists such data

The ismPlugin function extends the OpenSearch client by adding a namespace for managing index policies, allowing for easy interactions with the API to get, create, update, and delete policies, as well as handle associated operations like rollups and transforms.
https://github.com/opensearch-project/index-management-dashboards-plugin/blob/main/server/clusters/ism/ismPlugin.ts
In the ismPlugin they use API constant from "../../utils/constants" in which exists definition of parameter of Index to ISM configuration

export enum INDEX {
  OPENDISTRO_ISM_CONFIG = ".opendistro-ism-config",
}

This class document and control the mapping of policy schema in ISM Plugin.
Effectivily, after create a policy, I can search it in the .opendistro-ism-config index:

GET /.opendistro-ism-config/_search

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".opendistro-ism-config",
        "_id": "first_test",
        "_score": 1,
        "_source": {
          "policy": {
            "policy_id": "first_test",
            "description": "A first test of creation of an policy",
            "last_updated_time": 1736967823609,
            "schema_version": 21,
            "error_notification": null,
            "default_state": "pre_alias_removed",
            "states": [
              {
                "name": "pre_alias_removed",
                "actions": [
                  {
                    "retry": {
                      "count": 3,
                      "backoff": "exponential",
                      "delay": "1h"
                    },
                    "alias": {
                      "actions": [
                        {
                          "remove": {
                            "aliases": [
                              "commands"
                            ]
                          }
                        }
                      ]
                    }
                  }
                ],
                "transitions": []
              }
            ],
            "ism_template": [],
            "user": {
              "name": "admin",
              "backend_roles": [
                "admin"
              ],
              "roles": [
                "own_index",
                "all_access"
              ],
              "custom_attribute_names": [],
              "user_requested_tenant": null
            }
          }
        }
      }
    ]
  }
}

UPDATE

The alias to do the rollover is applied in the settings of index that we want to manage with the policy

PUT /our_index/_settings
{
  "index": {
    "plugins": {
      "index_state_management": {
        "rollover_alias": "name_of_alias_to_rollover"
      }
    }
  }
}

mcasas993 · 2025-01-21T21:08:19Z

Complete test of policy to rollover

1. Test in dashboard

### 1. Test in dashboard

Policy created to rollover alias

GET /.opendistro-ism-config/_search/
{
  "query": {
    "match": {
      "_id":"policy_rollover_6"
    }
  }
}

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".opendistro-ism-config",
        "_id": "policy_rollover_6",
        "_score": 1,
        "_source": {
          "policy": {
            "policy_id": "policy_rollover_6",
            "description": "Created first the policy",
            "last_updated_time": 1737399485357,
            "schema_version": 21,
            "error_notification": null,
            "default_state": "created",
            "states": [
              {
                "name": "created",
                "actions": [
                  {
                    "retry": {
                      "count": 3,
                      "backoff": "exponential",
                      "delay": "1m"
                    },
                    "rollover": {
                      "min_doc_count": 3,
                      "copy_alias": true
                    }
                  }
                ],
                "transitions": [
                  {
                    "state_name": "rollovered",
                    "conditions": {
                      "min_doc_count": 3
                    }
                  }
                ]
              },
              {
                "name": "rollovered",
                "actions": [],
                "transitions": []
              }
            ],
            "ism_template": [
              {
                "index_patterns": [
                  "another-*"
                ],
                "priority": 1,
                "last_updated_time": 1737382578280
              }
            ],
            "user": {
              "name": "admin",
              "backend_roles": [
                "admin"
              ],
              "roles": [
                "own_index",
                "all_access"
              ],
              "custom_attribute_names": [],
              "user_requested_tenant": null
            }
          }
        }
      }
    ]
  }
}

Index created and with an alias associated

Policy applied with the alias specified

GET /another-rollover/_settings

{
  "another-rollover-index-00001": {
    "settings": {
      "index": {
        "replication": {
          "type": "DOCUMENT"
        },
        "refresh_interval": "1s",
        "number_of_shards": "1",
        "plugins": {
          "index_state_management": {
            "rollover_alias": "another-rollover",
            "auto_manage": "false"
          }
        },
        "provided_name": "another-rollover-index-00001",
        "creation_date": "1737399216111",
        "number_of_replicas": "0",
        "uuid": "WSeQeDH7S7a9LQTs0qqTSw",
        "version": {
          "created": "136347827"
        }
      }
    }
  }
}

Create documents in the test index

Create five documents like this:

POST /another-rollover/_doc
{
  "name": "Another Example",
  "price": 19.99,
  "description": "We are such stuff as dreams are made on"
}

Results of the policy run

Result in another-rollover-index-00001 index

Result in another-rollover-index-00002 index

2. Test in manually in DevOp

Policy created to rollover alias

POST .opendistro-ism-config/_doc/manual_policy_rollover_3
     {
      "policy": {
          "description": "Created third manual rollover the policy",
          "last_updated_time": 1737399485357,
          "schema_version": 21,
          "error_notification": null,
          "default_state": "Initial",
          "states": [
            {
              "name": "Initial",
              "actions": [
                {
                  "retry": {
                    "count": 3,
                    "backoff": "exponential",
                    "delay": "1m"
                  },
                  "rollover": {
                    "min_doc_count": 3,
                    "copy_alias": true
                  }
                }
              ],
              "transitions": [
                {
                  "state_name": "Rollovered",
                  "conditions": {
                    "min_doc_count": 3
                  }
                }
              ]
            },
            {
              "name": "Rollovered",
              "actions": [],
              "transitions": []
            }
          ],
          "ism_template": [
            {
              "index_patterns": [
                "manual-*"
              ],
              "priority": 1,
              "last_updated_time": 1737382578280
            }
          ],
          "user": {
            "name": "admin",
            "backend_roles": [
              "admin"
            ],
            "roles": [
              "own_index",
              "all_access"
            ],
            "custom_attribute_names": [],
            "user_requested_tenant": null
          }
        }
      }

Index created and with an alias associated

Policy applied with the alias specified

PUT /manual-0001/_settings
{
  "index": {
    "plugins": {
      "index_state_management": {
        "rollover_alias": "manual"
      }
    }
  }
}

Policy applied with the alias specified

POST .opendistro-ism-config/_doc/
{
    "managed_index": {
      "name": "manual-0001",
      "enabled": false,
      "index": "manual-0001",
      "index_uuid": "OG3zRya0RoiKpjbxyJTbbA",
      "schedule": {
        "interval": {
          "start_time": 1737494828282,
          "period": 5,
          "unit": "Minutes"
        }
      },
      "last_updated_time": 1737496659085,
      "enabled_time": null,
      "policy_id": "manual_policy_rollover_3",
      "policy_seq_no": 2510,
      "policy_primary_term": 3,
      "policy": {
            "description": "Created third manual rollover the policy",
            "last_updated_time": 1737399485357,
            "schema_version": 21,
            "error_notification": null,
            "default_state": "Initial",
            "states": [
              {
                "name": "Initial",
                "actions": [
                  {
                    "retry": {
                      "count": 3,
                      "backoff": "exponential",
                      "delay": "1m"
                    },
                    "rollover": {
                      "min_doc_count": 3,
                      "copy_alias": true
                    }
                  }
                ],
                "transitions": [
                  {
                    "state_name": "Rollovered",
                    "conditions": {
                      "min_doc_count": 3
                    }
                  }
                ]
              },
              {
                "name": "Rollovered",
                "actions": [],
                "transitions": []
              }
            ],
            "ism_template": [
              {
                "index_patterns": [
                  "manual-*"
                ],
                "priority": 1,
                "last_updated_time": 1737382578280
              }
            ],
            "user": {
              "name": "admin",
              "backend_roles": [
                "admin"
              ],
              "roles": [
                "own_index",
                "all_access"
              ],
              "custom_attribute_names": [],
              "user_requested_tenant": null
            }
          },
      "change_policy": null,
      "jitter": 0.6
    }
}

Results of the policy run

Results of the policy run in manual-0001

mcasas993 · 2025-01-22T21:01:55Z

Complete test of policy to rollover based on the previous issue

Test the issue steps

Applied an ISM policy for rollover as follows:

Template modification

Edit /etc/filebeat/wazuh-template.json and add the following line inside the settings block:
```
```json
"index.plugins.index_state_management.rollover_alias": "test"
```
```
Restart wazuh-manager
```
systemctl restart wazuh-manager.service
```

ISM rollover and alias policy
- Push ISM policy to the Wazuh indexer cluster:
  
  "min_size": "250mb" for testing purposes only

curl -XPOST  -k -u admin:$admin_pass "https://127.0.0.1:9200/.opendistro-ism-config/_doc/MANUAL_wazuh_rollover_policy" -H 'Content-Type: application/json' -d'
{
    "policy": {
      "policy_id": "MANUAL_wazuh_rollover_policy",
      "description": "Wazuh rollover and alias policy created directly on index .opendistro-ism-config",
      "last_updated_time": 1737572429671,
      "schema_version": 21,
      "error_notification": null,
      "default_state": "active",
      "states": [
        {
          "name": "active",
          "actions": [
            {
              "retry": {
                "count": 3,
                "backoff": "exponential",
                "delay": "1m"
              },
              "rollover": {
                "min_size": "250mb",
                "copy_alias": false
              }
            }
          ],
          "transitions": []
        }
      ],
      "ism_template": [
        {
          "index_patterns": [
            "wazuh-alerts-*"
          ],
          "priority": 50,
          "last_updated_time": 1737572429671
        }
      ],
      "user": {
        "name": "admin",
        "backend_roles": [
          "admin"
        ],
        "roles": [
          "own_index",
          "all_access"
        ],
        "custom_attribute_names": [],
        "user_requested_tenant": null
      }
    }
}'

* Create initial index and quick-start the rolling process:
```bash
curl -k -u admin:$admin_pass -X PUT "https://127.0.0.1:9200/%3Ctest-1.x-%7Bnow%2Fd%7D-000001%3E?pretty" -H 'Content-Type: application/json' -d'

{
"aliases": {
"test": {
"is_write_index": true
}
}
}'

**RESULT**
Error: ""message": "Missing rollover_alias index setting [index=test-1.x-2025.01.22-000001]""

![Image](https://github.com/user-attachments/assets/3656f2b7-477a-49a9-88b7-05473217eaac)

3. Try with a the template to the index
   * Create a template to the index
    ![Image](https://github.com/user-attachments/assets/f731e246-fe23-4b65-8b86-31639e0d027f)
   
   *Delete the manage of the policy in the index
    ![Image](https://github.com/user-attachments/assets/dcaf651d-3970-4190-8c9d-2564b24eda91)

   *Delete the index
    In Dev Tools: DELETE /test-1.x-2025.01.22-000001

    * Create initial index and quick-start the rolling process:
    ```bash
    curl -k -u admin:$admin_pass -X PUT "https://127.0.0.1:9200/%3Ctest-1.x-%7Bnow%2Fd%7D-000001%3E?pretty" -H 'Content-Type: application/json' -d'
{
  "aliases": {
    "test": {
      "is_write_index": true
    }
  }
}'

RESULT
Error: ""message": "Missing rollover_alias index setting [index=test-1.x-2025.01.22-000001]""

Put the settiing in the index

PUT /test/_settings
{
  "index": {
    "plugins": {
      "index_state_management": {
        "rollover_alias": "test"
      }
    }
  }
}

RESULT
Error: ""message": "{
"cause": "Rollover alias [test] can point to multiple indices, found duplicated alias [[test]] in index template [test]",
"message": "Failed to rollover index [index=test-1.x-2025.01.22-000001]"
}""

AlexRuiz7 · 2025-01-24T16:46:50Z

Simple ISM configuration

Below there is a simple ISM configuration to automatically roll over the wazuh-commands indices.

PUT _template/wazuh-commands
{
  "index_patterns": [
    "wazuh-commands*"
  ],
  "mappings": {
    "date_detection": false,
    "dynamic": "true",
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "agent": {
        "properties": {
          "groups": {
            "ignore_above": 1024,
            "type": "keyword"
          }
        }
      },
      "command": {
        "properties": {
          "action": {
            "properties": {
              "args": {
                "type": "object"
              },
              "name": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "version": {
                "ignore_above": 1024,
                "type": "keyword"
              }
            }
          },
          "order_id": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "request_id": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "result": {
            "properties": {
              "code": {
                "type": "short"
              },
              "data": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "message": {
                "ignore_above": 1024,
                "type": "keyword"
              }
            }
          },
          "source": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "status": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "target": {
            "properties": {
              "id": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "type": {
                "ignore_above": 1024,
                "type": "keyword"
              }
            }
          },
          "timeout": {
            "type": "short"
          },
          "user": {
            "ignore_above": 1024,
            "type": "keyword"
          }
        }
      },
      "delivery_timestamp": {
        "type": "date"
      }
    }
  },
  "order": 1,
  "settings": {
    "index": {
      "number_of_replicas": "0",
      "number_of_shards": "1",
      "query.default_field": [
        "command.source",
        "command.target.type",
        "command.status",
        "command.action.name"
      ],
      "refresh_interval": "5s",
      "plugins.index_state_management.rollover_alias": "wazuh-commands"
    }
  },
  "version": 500
}

PUT _plugins/_ism/policies/wazuh_rollover_policy
{
  "policy": {
    "description": "Wazuh rollover and alias policy",
    "default_state": "active",
    "states": [
      {
        "name": "active",
        "actions": [
          {
            "rollover": {
              "min_index_age": "5m"
            }
          }
        ]
      }
    ],
    "ism_template": {
      "index_patterns": ["wazuh-commands*"],
      "priority": "50"
    }
  }
}

PUT wazuh-commands-0001
{
  "aliases": {
    "wazuh-commands": {
      "is_write_index": true
    }
  }
}

mcasas993 · 2025-01-30T17:21:34Z

Test ISM rollover configuration by creating the policy directly on the index `.opendistro-ism-config`

Add index template for the indices with the setting "plugins.index_state_management.rollover_alias": "<alias>".

Upload template

PUT _template/wazuh-commands
{
  "index_patterns": [
    "wazuh-commands*"
  ],
  "mappings": {
    "date_detection": false,
    "dynamic": "true",
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "agent": {
        "properties": {
          "groups": {
            "ignore_above": 1024,
            "type": "keyword"
          }
        }
      },
      "command": {
        "properties": {
          "action": {
            "properties": {
              "args": {
                "type": "object"
              },
              "name": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "version": {
                "ignore_above": 1024,
                "type": "keyword"
              }
            }
          },
          "order_id": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "request_id": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "result": {
            "properties": {
              "code": {
                "type": "short"
              },
              "data": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "message": {
                "ignore_above": 1024,
                "type": "keyword"
              }
            }
          },
          "source": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "status": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "target": {
            "properties": {
              "id": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "type": {
                "ignore_above": 1024,
                "type": "keyword"
              }
            }
          },
          "timeout": {
            "type": "short"
          },
          "user": {
            "ignore_above": 1024,
            "type": "keyword"
          }
        }
      },
      "delivery_timestamp": {
        "type": "date"
      }
    }
  },
  "order": 1,
  "settings": {
    "index": {
      "number_of_replicas": "0",
      "number_of_shards": "1",
      "query.default_field": [
        "command.source",
        "command.target.type",
        "command.status",
        "command.action.name"
      ],
      "refresh_interval": "5s",
      "plugins.index_state_management.rollover_alias": "wazuh-commands"
    }
  },
  "version": 500
}

Index policy

Index policy in `.opendistro-ism-config`

POST .opendistro-ism-config/_doc/wazuh_rollover_policy
{
          "policy": {
            "policy_id": "wazuh_rollover_policy",
            "description": "Wazuh rollover and alias policy",
            "last_updated_time": 1738255639727,
            "schema_version": 21,
            "error_notification": null,
            "default_state": "active",
            "states": [
              {
                "name": "active",
                "actions": [
                  {
                    "retry": {
                      "count": 3,
                      "backoff": "exponential",
                      "delay": "1m"
                    },
                    "rollover": {
                      "min_index_age": "5m",
                      "copy_alias": false
                    }
                  }
                ],
                "transitions": []
              }
            ],
            "ism_template": [
              {
                "index_patterns": [
                  "wazuh-commands*"
                ],
                "priority": 50,
                "last_updated_time": 1738255639727
              }
            ],
            "user": {
              "name": "admin",
              "backend_roles": [
                "admin"
              ],
              "roles": [
                "own_index",
                "all_access"
              ],
              "custom_attribute_names": [],
              "user_requested_tenant": null
            }
          }
      }

Create initial index with alias

Create index alias

PUT wazuh-commands-0001
{
  "aliases": {
    "wazuh-commands": {
      "is_write_index": true
    }
  }
}

RESULTS

Policy created

Policy managed indexes

Index created

Configuration of active index with alias set

mcasas993 · 2025-01-30T23:55:26Z

Define aliases name

The best options to the alias are the name for the index without any aggregation of number, date or anything:

wazuh-alerts
wazuh-commands

Because the prefix "wazuh" indicates that these are indexes generated for Wazuh automatically.

mcasas993 · 2025-01-31T00:09:19Z

Define rollover policies

After careful consideration and analysis of the requirements from both the previous issues and industry best practices, I think the best options to implement a rollover policy for Wazuh data streams are the following conditions:

Maximum Document Count: 20,000,000,000 documents
Maximum Shard Size: 25 GB per shard
Maximum Index Age: 7 days

Justification:

Maximum Document Count: Setting a maximum document count of 20 billion ensures we adhere to the limitations of OpenSearch while accommodating hyper-scaling scenarios.
Maximum Shard Size: A maximum shard size of 25 GB aligns with best practices for performance. Shard sizes in this range (between 10-50 GB) help balance the need for efficient metadata management while allowing for effective data distribution across the cluster. This approach mitigates issues related to excessive small shards, which could lead to exhausting JVM memory.
Maximum Index Age: Limiting the index age to 7 days prevents the cluster from being cluttered with indices that do not, in fact, enhance performance. If the shard size limit or document count limit has not been reached, there is not much benefit in generating numerous indexes just because of their age.

Policies

Policy for `wazuh-commands*`

"policy": {
    "policy_id": "wazuh-command_rollover_policy",
    "description": "Wazuh-command rollover and alias policy",
    "last_updated_time": XXXXX,
    "schema_version": XXXX,
    "error_notification": null,
    "default_state": "active",
    "states": [
      {
        "name": "active",
        "actions": [
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "1m"
            },
            "rollover": {
              "min_doc_count": 200000000,
              "min_index_age": "7d",
              "min_primary_shard_size": "25gb",
              "copy_alias": false
            }
          }
        ],
        "transitions": []
      }
    ],
    "ism_template": [
      {
        "index_patterns": [
          "wazuh-commands*"
        ],
        "priority": 50,
        "last_updated_time": XXXXX
      }
    ]
  }

Policy for `wazuh-alerts*`

"policy": {
    "policy_id": "wazuh-alerts_rollover_policy",
    "description": "Wazuh-alerts rollover and alias policy",
    "last_updated_time": XXXXX,
    "schema_version": XXXX,
    "error_notification": null,
    "default_state": "active",
    "states": [
      {
        "name": "active",
        "actions": [
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "1m"
            },
            "rollover": {
              "min_doc_count": 200000000,
              "min_index_age": "7d",
              "min_primary_shard_size": "25gb",
              "copy_alias": false
            }
          }
        ],
        "transitions": []
      }
    ],
    "ism_template": [
      {
        "index_patterns": [
          "wazuh-alerts*"
        ],
        "priority": 50,
        "last_updated_time": XXXXX
      }
    ]
  }

f-galland · 2025-02-07T19:31:01Z

We were able to get the policy to be applied from our setup plugin:

Add a rollover policy to the setup plugin wazuh-indexer-plugins#269

Comments in the PR explain how to try that out.

mcasas993 · 2025-02-12T17:28:16Z

Like we can see in the tests mentioned in the last comment, we checked that:

[1] We can index the rollover policy directly in the .opendistro-ism-config, but as the index was not initialized by the ISM plugin, it was lacking the template and the mappings, causing it to fail afterward, so we included the template from the ISM plugin.
[2] As a result, the policy works correctly, and the new indices are managed by it, performing the rollover of the indices.
[3] The policy is recognized by the ISM plugin of OpenSearch.

The solution works, but we would to overlap the ISM plugin responsibilities, creating the index template it uses for its internal index. We are unaware of the possible collateral effects of this, and we'll also need to maintain that index template, leading to potential bugs if the template is updated on the ISM plugin side.

AlexRuiz7 · 2025-02-14T14:27:53Z

Checkpoint

There are currently 3 possible alternatives:

The proposed approach on this issue (Rollover and alias for stream indices #591 (comment)).
Risks: commented above.
Extending the setup plugin to perform HTTP requests to the ISM API to create the policy.
Risks: it could be possible that a document is pushed to one of the managed indices (wazuh-alerts or wazuh-commands) before the policy is created, if the request to the API takes more time than expected.
Fork the ISM plugin and extend it so it creates the policy on start.
Risks: we will have to maintain a new fork repository. Kotlin.

AlexRuiz7 · 2025-02-17T15:53:36Z

We have been discussing the alternatives, and we are finally aiming with the first alternative.

Before that, we are going to do a deep research about the index and all the operations the plugin performs over it (the more we know about the index and its possible transforms the better), and check that the ISM plugin keeps working properly (all its supported features) while the index is being created by our setup plugin.

AlexRuiz7 added level/task Task issue type/enhancement Enhancement issue labels Dec 11, 2024

AlexRuiz7 mentioned this issue Dec 11, 2024

Data persistence model redesign wazuh/wazuh#22887

Open

25 tasks

wazuhci added this to XDR+SIEM/Release 5.0.0 Dec 11, 2024

wazuhci moved this to Backlog in XDR+SIEM/Release 5.0.0 Dec 11, 2024

AlexRuiz7 assigned mcasas993 Jan 14, 2025

wazuhci moved this from Backlog to In progress in XDR+SIEM/Release 5.0.0 Jan 14, 2025

AlexRuiz7 mentioned this issue Jan 15, 2025

Explore how to use the ISM features from another plugin wazuh/wazuh-indexer-plugins#12

Closed

5 tasks

AlexRuiz7 mentioned this issue Jan 23, 2025

Make agents and commands indices visible wazuh/wazuh-indexer-plugins#240

Closed

4 tasks

wazuhci moved this from In progress to On hold in XDR+SIEM/Release 5.0.0 Jan 23, 2025

AlexRuiz7 self-assigned this Jan 24, 2025

wazuhci moved this from On hold to In progress in XDR+SIEM/Release 5.0.0 Jan 24, 2025

AlexRuiz7 removed their assignment Jan 24, 2025

wazuhci moved this from In progress to On hold in XDR+SIEM/Release 5.0.0 Jan 24, 2025

wazuhci moved this from On hold to In progress in XDR+SIEM/Release 5.0.0 Jan 27, 2025

wazuhci moved this from In progress to On hold in XDR+SIEM/Release 5.0.0 Jan 29, 2025

wazuhci moved this from On hold to In progress in XDR+SIEM/Release 5.0.0 Jan 30, 2025

wazuhci moved this from In progress to Pending review in XDR+SIEM/Release 5.0.0 Jan 31, 2025

wazuhci moved this from Pending review to In review in XDR+SIEM/Release 5.0.0 Feb 6, 2025

wazuhci moved this from In review to In progress in XDR+SIEM/Release 5.0.0 Feb 6, 2025

f-galland linked a pull request Feb 7, 2025 that will close this issue

Add a rollover policy to the setup plugin wazuh/wazuh-indexer-plugins#269

Draft

wazuhci moved this from In progress to Blocked in XDR+SIEM/Release 5.0.0 Feb 14, 2025

wazuhci moved this from Blocked to On hold in XDR+SIEM/Release 5.0.0 Feb 17, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rollover and alias for stream indices #591

Rollover and alias for stream indices #591

AlexRuiz7 commented Dec 11, 2024 •

edited

Loading

mcasas993 commented Jan 16, 2025 •

edited

Loading

mcasas993 commented Jan 21, 2025 •

edited

Loading

Policy created to rollover alias

Index created and with an alias associated

Policy applied with the alias specified

Create documents in the test index

Results of the policy run

Result in another-rollover-index-00001 index

Result in another-rollover-index-00002 index

2. Test in manually in DevOp

Policy created to rollover alias

Index created and with an alias associated

Policy applied with the alias specified

Policy applied with the alias specified

Results of the policy run

Results of the policy run in manual-0001

mcasas993 commented Jan 22, 2025 •

edited

Loading

Test the issue steps

AlexRuiz7 commented Jan 24, 2025 •

edited

Loading

mcasas993 commented Jan 30, 2025 •

edited by AlexRuiz7

Loading

mcasas993 commented Jan 30, 2025 •

edited

Loading

mcasas993 commented Jan 31, 2025 •

edited by AlexRuiz7

Loading

f-galland commented Feb 7, 2025

mcasas993 commented Feb 12, 2025 •

edited by AlexRuiz7

Loading

AlexRuiz7 commented Feb 14, 2025

AlexRuiz7 commented Feb 17, 2025

Rollover and alias for stream indices #591

Rollover and alias for stream indices #591

Comments

AlexRuiz7 commented Dec 11, 2024 • edited Loading

Description

Functional requirements

Implementation restrictions

Plan

mcasas993 commented Jan 16, 2025 • edited Loading

Investigation of how the ISM plugin persists such data

UPDATE

mcasas993 commented Jan 21, 2025 • edited Loading

Complete test of policy to rollover

Policy created to rollover alias

Index created and with an alias associated

Policy applied with the alias specified

Create documents in the test index

Results of the policy run

Result in another-rollover-index-00001 index

Result in another-rollover-index-00002 index

2. Test in manually in DevOp

Policy created to rollover alias

Index created and with an alias associated

Policy applied with the alias specified

Policy applied with the alias specified

Results of the policy run

Results of the policy run in manual-0001

mcasas993 commented Jan 22, 2025 • edited Loading

Complete test of policy to rollover based on the previous issue

Test the issue steps

AlexRuiz7 commented Jan 24, 2025 • edited Loading

Simple ISM configuration

mcasas993 commented Jan 30, 2025 • edited by AlexRuiz7 Loading

Test ISM rollover configuration by creating the policy directly on the index .opendistro-ism-config

RESULTS

Policy created

Policy managed indexes

Index created

Configuration of active index with alias set

mcasas993 commented Jan 30, 2025 • edited Loading

Define aliases name

mcasas993 commented Jan 31, 2025 • edited by AlexRuiz7 Loading

Define rollover policies

Justification:

Policies

Policy for wazuh-commands*

Policy for wazuh-alerts*

f-galland commented Feb 7, 2025

mcasas993 commented Feb 12, 2025 • edited by AlexRuiz7 Loading

AlexRuiz7 commented Feb 14, 2025

Checkpoint

AlexRuiz7 commented Feb 17, 2025

AlexRuiz7 commented Dec 11, 2024 •

edited

Loading

mcasas993 commented Jan 16, 2025 •

edited

Loading

mcasas993 commented Jan 21, 2025 •

edited

Loading

mcasas993 commented Jan 22, 2025 •

edited

Loading

AlexRuiz7 commented Jan 24, 2025 •

edited

Loading

mcasas993 commented Jan 30, 2025 •

edited by AlexRuiz7

Loading

Test ISM rollover configuration by creating the policy directly on the index `.opendistro-ism-config`

mcasas993 commented Jan 30, 2025 •

edited

Loading

mcasas993 commented Jan 31, 2025 •

edited by AlexRuiz7

Loading

Policy for `wazuh-commands*`

Policy for `wazuh-alerts*`

mcasas993 commented Feb 12, 2025 •

edited by AlexRuiz7

Loading