Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] The Wazuh indexer fails to start during an upgrade if the service was active #723

Open
rauldpm opened this issue Feb 27, 2025 · 4 comments · Fixed by #725
Open

[BUG] The Wazuh indexer fails to start during an upgrade if the service was active #723

rauldpm opened this issue Feb 27, 2025 · 4 comments · Fixed by #725
Assignees
Labels
level/task Task issue type/bug Bug issue

Comments

@rauldpm
Copy link
Member

rauldpm commented Feb 27, 2025

Describe the bug

The Wazuh indexer upgrade seems to try to restart the Wazuh indexer component using systemctl but the status after the upgrade is dead


===================================================================================================================================================================================================================
 Package                                          Arch                                      Version                                        Repository                                                         Size
===================================================================================================================================================================================================================
Updating:
 wazuh-indexer                                    x86_64                                    4.11.0-1                                       /wazuh-indexer-4.11.0-1.x86_64                                    1.0 G

Transaction Summary
===================================================================================================================================================================================================================
Upgrade  1 Package

Total size: 1.0 G
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Stop existing wazuh-indexer.service
  Updating   : wazuh-indexer-4.11.0-1.x86_64                                                                                                                                                                   1/2 
Restarting wazuh-indexer service...
Stop existing wazuh-indexer.service
  Cleanup    : wazuh-indexer-4.10.1-1.x86_64                                                                                                                                                                   2/2 
  Verifying  : wazuh-indexer-4.11.0-1.x86_64                                                                                                                                                                   1/2 
  Verifying  : wazuh-indexer-4.10.1-1.x86_64                                                                                                                                                                   2/2 

Updated:
  wazuh-indexer.x86_64 0:4.11.0-1                                                                                                                                                                                  

Complete!
[root@centos7 vagrant]# systemctl status wazuh-indexer
● wazuh-indexer.service - wazuh-indexer
   Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Thu 2025-02-27 21:39:03 UTC; 21s ago
     Docs: https://documentation.wazuh.com
  Process: 4178 ExecStart=/usr/share/wazuh-indexer/bin/systemd-entrypoint -p ${PID_DIR}/wazuh-indexer.pid --quiet (code=exited, status=143)
 Main PID: 4178 (code=exited, status=143)

If the service is stopped before the upgrade, the Process reference does not appear

Looking at the SPEC file, it seems that there is a problem in the post stage https://github.com/wazuh/wazuh-indexer/blob/v4.11.0/distribution/packages/src/rpm/wazuh-indexer.rpm.spec#L208-L215 as the restart command should keep the service active

Based on the Debian test done in this comment, this is an unexpected behavior

To Reproduce
Steps to reproduce the behavior:

  1. Install a 4.10.1 Wazuh indexer and configure the certificates
  2. Start the Wazuh indexer
  3. Initialize the cluster
  4. Upgrade to 4.11.0 without stopping the service
  5. Check status

Expected behavior
If the package should keep the service active, it should be started correctly by the package like the Debian package does

Plugins
N/A

Screenshots
N/A

Host/Environment (please complete the following information):

  • OS: CentOS
  • Version 7

Additional context
N/A

@rauldpm rauldpm added level/task Task issue type/bug Bug issue labels Feb 27, 2025
@f-galland f-galland self-assigned this Feb 28, 2025
@wazuhci wazuhci moved this to In progress in XDR+SIEM/Release 4.11.1 Feb 28, 2025
@f-galland
Copy link
Member

I've added an echo command at the start of each phase of the .spec script, and it seems the %preun is executed at the end of the upgrade process of the package:

[root@alma9 ~]# yum reinstall -y ./wazuh-indexer-4.11.1.x86_64.rpm 
Last metadata expiration check: 1:15:46 ago on Fri Feb 28 12:55:37 2025.
Dependencies resolved.
======================================================================================================================================================================================
 Package                                        Architecture                            Version                                   Repository                                     Size
======================================================================================================================================================================================
Reinstalling:
 wazuh-indexer                                  x86_64                                  4.11.1-0                                  @commandline                                  831 M

Transaction Summary
======================================================================================================================================================================================

Total size: 831 M
Installed size: 1.0 G
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                              1/1 
  Running scriptlet: wazuh-indexer-4.11.1-0.x86_64                                                                                                                                1/2 
%pre stage
Stop existing wazuh-indexer.service
line 164

  Reinstalling     : wazuh-indexer-4.11.1-0.x86_64                                                                                                                                1/2 
  Running scriptlet: wazuh-indexer-4.11.1-0.x86_64                                                                                                                                1/2 
%post stage
Restarting wazuh-indexer service...

  Running scriptlet: wazuh-indexer-4.11.1-0.x86_64                                                                                                                                2/2 
%preun stage
Stop existing wazuh-indexer.service
line 232

  Cleanup          : wazuh-indexer-4.11.1-0.x86_64                                                                                                                                2/2 
  Running scriptlet: wazuh-indexer-4.11.1-0.x86_64                                                                                                                                2/2 
  Verifying        : wazuh-indexer-4.11.1-0.x86_64                                                                                                                                1/2 
  Verifying        : wazuh-indexer-4.11.1-0.x86_64                                                                                                                                2/2 

Reinstalled:
  wazuh-indexer-4.11.1-0.x86_64                                                                                                                                                       

Complete!

@f-galland
Copy link
Member

f-galland commented Feb 28, 2025

The above is confirmed by the following source:

Although I haven't been able to back this with official documentation.

I found official documentation to back this up:

For reference, here's the order in which scripts are executed on a single
package upgrade:

  all-%pretrans
  ...
  any-%triggerprein (%triggerprein from other packages set off by new install)
  new-%triggerprein
  new-%pre	for new version of package being installed
  ...		(all new files are installed)
  new-%post	for new version of package being installed

  any-%triggerin (%triggerin from other packages set off by new install)
  new-%triggerin
  old-%triggerun
  any-%triggerun (%triggerun from other packages set off by old uninstall)

  old-%preun	for old version of package being removed
  ...		(all old files are removed)
  old-%postun	for old version of package being removed

  old-%triggerpostun
  any-%triggerpostun (%triggerpostun from other packages set off by old un
		install)
  ...
  all-%posttrans

@f-galland
Copy link
Member

f-galland commented Feb 28, 2025

The %preun scriptlet that gets to run is the one in the old package, instead of the one being installed.

I'm not sure we can make this work for upgrades until we have deployed 2 consecutive versions with the fix.

In order to test this assertion, I installed 4.10 from the repos, and upgraded to a custom package with an echo statement in the %preun phase:

$ grep -A3 '^%preun' distribution/packages/src/rpm/wazuh-indexer.rpm.spec 
%preun
set -e

echo "Running a custom %preun phase"
Upgrading to this custom package doesn't show my custom message
[root@alma9 ~]# yum install ./wazuh-indexer-4.11.1.x86_64.rpm 
Last metadata expiration check: 1:01:24 ago on Fri Feb 28 15:58:29 2025.
Dependencies resolved.
======================================================================================================================================================================================
 Package                                        Architecture                            Version                                   Repository                                     Size
======================================================================================================================================================================================
Upgrading:
 wazuh-indexer                                  x86_64                                  4.11.1-0                                  @commandline                                  831 M

Transaction Summary
======================================================================================================================================================================================
Upgrade  1 Package

Total size: 831 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                              1/1 
  Running scriptlet: wazuh-indexer-4.11.1-0.x86_64                                                                                                                                1/2 
Stop existing wazuh-indexer.service

  Upgrading        : wazuh-indexer-4.11.1-0.x86_64                                                                                                                                1/2 
  Running scriptlet: wazuh-indexer-4.11.1-0.x86_64                                                                                                                                1/2 
  Running scriptlet: wazuh-indexer-4.10.1-1.x86_64                                                                                                                                2/2 
  Cleanup          : wazuh-indexer-4.10.1-1.x86_64                                                                                                                                2/2 
  Running scriptlet: wazuh-indexer-4.10.1-1.x86_64                                                                                                                                2/2 
  Verifying        : wazuh-indexer-4.11.1-0.x86_64                                                                                                                                1/2 
  Verifying        : wazuh-indexer-4.10.1-1.x86_64                                                                                                                                2/2 

Upgraded:
  wazuh-indexer-4.11.1-0.x86_64                                                                                                                                                       

Complete!
The message does in fact show when upgrading from the custom 4.11 package.
[root@alma9 ~]# yum reinstall -y ./wazuh-indexer-4.11.1.x86_64.rpm 
Last metadata expiration check: 2:13:08 ago on Fri Feb 28 15:58:29 2025.
Dependencies resolved.
======================================================================================================================================================================================
 Package                                        Architecture                            Version                                   Repository                                     Size
======================================================================================================================================================================================
Reinstalling:
 wazuh-indexer                                  x86_64                                  4.11.1-0                                  @commandline                                  831 M

Transaction Summary
======================================================================================================================================================================================

Total size: 831 M
Installed size: 1.0 G
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                              1/1 
  Running scriptlet: wazuh-indexer-4.11.1-0.x86_64                                                                                                                                1/2 
Stop existing wazuh-indexer.service

  Reinstalling     : wazuh-indexer-4.11.1-0.x86_64                                                                                                                                1/2 
  Running scriptlet: wazuh-indexer-4.11.1-0.x86_64                                                                                                                                1/2 
  Running scriptlet: wazuh-indexer-4.11.1-0.x86_64                                                                                                                                2/2 
Running a custom %preun phase
Restarting wazuh-indexer service...

  Cleanup          : wazuh-indexer-4.11.1-0.x86_64                                                                                                                                2/2 
  Running scriptlet: wazuh-indexer-4.11.1-0.x86_64                                                                                                                                2/2 
  Verifying        : wazuh-indexer-4.11.1-0.x86_64                                                                                                                                1/2 
  Verifying        : wazuh-indexer-4.11.1-0.x86_64                                                                                                                                2/2 

Reinstalled:
  wazuh-indexer-4.11.1-0.x86_64                                                                                                                                                       

Complete!

@f-galland f-galland linked a pull request Feb 28, 2025 that will close this issue
3 tasks
@wazuhci wazuhci moved this from In progress to Pending review in XDR+SIEM/Release 4.11.1 Feb 28, 2025
@f-galland
Copy link
Member

We need to check whether these changes need to be ported to 5.0.0 and 4.12.0.

@wazuhci wazuhci moved this from Pending review to On hold in XDR+SIEM/Release 4.11.1 Feb 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
level/task Task issue type/bug Bug issue
Projects
Status: On hold
Development

Successfully merging a pull request may close this issue.

2 participants