Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Upgrade to 4.11.0 fails with: /var/lib/wazuh-indexer/tmp/wazuh-indexer.restart: No such file or directory #721

Open
f-galland opened this issue Feb 27, 2025 · 2 comments · Fixed by #722 · May be fixed by #724
Open
Assignees
Labels
level/task Task issue type/bug Bug issue

Comments

@f-galland
Copy link
Member

Description

Upgrading to 4.11.0 from <4.11.0 fails for RPM-based:

Stop existing wazuh-indexer.service
touch: cannot touch '/var/lib/wazuh-indexer/tmp/wazuh-indexer.restart': No such file or directory
error: %prein(wazuh-indexer-4.11.0-1.x86_64) scriptlet failed, exit status 1
Error in PREIN scriptlet in rpm package wazuh-indexer
Verifying : wazuh-indexer-4.11.0-1.x86_64 1/2
Verifying : wazuh-indexer-4.10.1-1.x86_64 2/2
Failed:
wazuh-indexer-4.10.1-1.x86_64 wazuh-indexer-4.11.0-1.x86_64

Steps to reproduce

  • Install wazuh-indexer 4.10.1.
  • Upgrade to 4.11.0

Environment

  • OS: RPM-based
@f-galland f-galland added level/task Task issue type/bug Bug issue labels Feb 27, 2025
@f-galland
Copy link
Member Author

f-galland commented Feb 27, 2025

The issue is caused by 4.10.1 packages not including the /var/lib/wazuh-indexer/tmp directory:

[root@alma9 ~]# rpm2cpio wazuh-indexer-4.10.1-1.x86_64.rpm | cpio -idmv 2>&1 /dev/null
2151935 blocks
[root@alma9 ~]# ls var/lib/wazuh-indexer/tmp
ls: cannot access 'var/lib/wazuh-indexer/tmp': No such file or directory
[root@alma9 ~]# ls var/lib/wazuh-indexer/
performance_analyzer_enabled.conf  rca_enabled.conf

4.11.0 packages expect that directory to be present at the %pre (pre-install) phase:

@rauldpm
Copy link
Member

rauldpm commented Feb 27, 2025

After conducting research in a huddle with @f-galland, we found the following:

  1. If the 4.10.1 package is installed and the service is not started, the upgrade is successful.
  2. If the 4.10.1 package is installed and the service is started, the upgrade will fail if it is performed without stopping the service.
  3. If the 4.10.1 package is installed and the service is started, the upgrade is successful if it is performed according to the documentation (the indexer service is stopped).
Upgrade following the documentation 🟢
[root@centos7 vagrant]# systemctl status wazuh-indexer
● wazuh-indexer.service - wazuh-indexer
   Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2025-02-27 19:52:31 UTC; 3s ago
     Docs: https://documentation.wazuh.com
 Main PID: 3828 (java)
   CGroup: /system.slice/wazuh-indexer.service
           └─3828 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=t...

Feb 27 19:52:25 centos7 systemd-entrypoint[3828]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.16.0.jar)
Feb 27 19:52:25 centos7 systemd-entrypoint[3828]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Feb 27 19:52:25 centos7 systemd-entrypoint[3828]: WARNING: System::setSecurityManager will be removed in a future release
Feb 27 19:52:25 centos7 systemd-entrypoint[3828]: Feb 27, 2025 7:52:25 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
Feb 27 19:52:25 centos7 systemd-entrypoint[3828]: WARNING: COMPAT locale provider will be removed in a future release
Feb 27 19:52:25 centos7 systemd-entrypoint[3828]: WARNING: A terminally deprecated method in java.lang.System has been called
Feb 27 19:52:25 centos7 systemd-entrypoint[3828]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.16.0.jar)
Feb 27 19:52:25 centos7 systemd-entrypoint[3828]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Feb 27 19:52:25 centos7 systemd-entrypoint[3828]: WARNING: System::setSecurityManager will be removed in a future release
Feb 27 19:52:31 centos7 systemd[1]: Started wazuh-indexer.
[root@centos7 vagrant]# systemctl stop wazuh-indexer

[root@centos7 vagrant]# yum upgrade wazuh-indexer -y
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
---> Package wazuh-indexer.x86_64 0:4.10.1-1 will be updated
---> Package wazuh-indexer.x86_64 0:4.11.0-1 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

===================================================================================================================================================================================================================
 Package                                                 Arch                                             Version                                            Repository                                       Size
===================================================================================================================================================================================================================
Updating:
 wazuh-indexer                                           x86_64                                           4.11.0-1                                           wazuh                                           831 M

Transaction Summary
===================================================================================================================================================================================================================
Upgrade  1 Package

Total download size: 831 M
Downloading packages:
wazuh-indexer-4.11.0-1.x86_64.rpm                                                                                                                                                           | 831 MB  00:00:42     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : wazuh-indexer-4.11.0-1.x86_64                                                                                                                                                                   1/2 
### NOT starting on installation, please execute the following statements to configure wazuh-indexer service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable wazuh-indexer.service
### You can start wazuh-indexer service by executing
 sudo systemctl start wazuh-indexer.service
  Cleanup    : wazuh-indexer-4.10.1-1.x86_64                                                                                                                                                                   2/2 
  Verifying  : wazuh-indexer-4.11.0-1.x86_64                                                                                                                                                                   1/2 
  Verifying  : wazuh-indexer-4.10.1-1.x86_64                                                                                                                                                                   2/2 

Updated:
  wazuh-indexer.x86_64 0:4.11.0-1                                                                                                                                                                                  

Complete!
[root@centos7 vagrant]# systemctl status wazuh-indexer
● wazuh-indexer.service - wazuh-indexer
   Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: https://documentation.wazuh.com

Feb 27 19:52:25 centos7 systemd-entrypoint[3828]: WARNING: System::setSecurityManager will be removed in a future release
Feb 27 19:52:25 centos7 systemd-entrypoint[3828]: Feb 27, 2025 7:52:25 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
Feb 27 19:52:25 centos7 systemd-entrypoint[3828]: WARNING: COMPAT locale provider will be removed in a future release
Feb 27 19:52:25 centos7 systemd-entrypoint[3828]: WARNING: A terminally deprecated method in java.lang.System has been called
Feb 27 19:52:25 centos7 systemd-entrypoint[3828]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.16.0.jar)
Feb 27 19:52:25 centos7 systemd-entrypoint[3828]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Feb 27 19:52:25 centos7 systemd-entrypoint[3828]: WARNING: System::setSecurityManager will be removed in a future release
Feb 27 19:52:31 centos7 systemd[1]: Started wazuh-indexer.
Feb 27 19:52:36 centos7 systemd[1]: Stopping wazuh-indexer...
Feb 27 19:52:36 centos7 systemd[1]: Stopped wazuh-indexer.
[root@centos7 vagrant]# systemctl start wazuh-indexer
[root@centos7 vagrant]# systemctl status wazuh-indexer
● wazuh-indexer.service - wazuh-indexer
   Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2025-02-27 19:54:41 UTC; 1s ago
     Docs: https://documentation.wazuh.com
 Main PID: 4143 (java)
   CGroup: /system.slice/wazuh-indexer.service
           └─4143 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=t...

Feb 27 19:54:35 centos7 systemd-entrypoint[4143]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.16.0.jar)
Feb 27 19:54:35 centos7 systemd-entrypoint[4143]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Feb 27 19:54:35 centos7 systemd-entrypoint[4143]: WARNING: System::setSecurityManager will be removed in a future release
Feb 27 19:54:36 centos7 systemd-entrypoint[4143]: Feb 27, 2025 7:54:36 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
Feb 27 19:54:36 centos7 systemd-entrypoint[4143]: WARNING: COMPAT locale provider will be removed in a future release
Feb 27 19:54:36 centos7 systemd-entrypoint[4143]: WARNING: A terminally deprecated method in java.lang.System has been called
Feb 27 19:54:36 centos7 systemd-entrypoint[4143]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.16.0.jar)
Feb 27 19:54:36 centos7 systemd-entrypoint[4143]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Feb 27 19:54:36 centos7 systemd-entrypoint[4143]: WARNING: System::setSecurityManager will be removed in a future release
Feb 27 19:54:41 centos7 systemd[1]: Started wazuh-indexer.

Upgrade without stopping the service 🔴 (without following the documentation)
[root@centos7 vagrant]# systemctl status wazuh-indexer
● wazuh-indexer.service - wazuh-indexer
   Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2025-02-27 20:05:22 UTC; 8s ago
     Docs: https://documentation.wazuh.com
 Main PID: 4499 (java)
   CGroup: /system.slice/wazuh-indexer.service
           └─4499 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=t...

Feb 27 20:05:16 centos7 systemd-entrypoint[4499]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.16.0.jar)
Feb 27 20:05:16 centos7 systemd-entrypoint[4499]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Feb 27 20:05:16 centos7 systemd-entrypoint[4499]: WARNING: System::setSecurityManager will be removed in a future release
Feb 27 20:05:16 centos7 systemd-entrypoint[4499]: Feb 27, 2025 8:05:16 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
Feb 27 20:05:16 centos7 systemd-entrypoint[4499]: WARNING: COMPAT locale provider will be removed in a future release
Feb 27 20:05:17 centos7 systemd-entrypoint[4499]: WARNING: A terminally deprecated method in java.lang.System has been called
Feb 27 20:05:17 centos7 systemd-entrypoint[4499]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.16.0.jar)
Feb 27 20:05:17 centos7 systemd-entrypoint[4499]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Feb 27 20:05:17 centos7 systemd-entrypoint[4499]: WARNING: System::setSecurityManager will be removed in a future release
Feb 27 20:05:22 centos7 systemd[1]: Started wazuh-indexer.
[root@centos7 vagrant]# yum upgrade wazuh-indexer
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
---> Package wazuh-indexer.x86_64 0:4.10.1-1 will be updated
---> Package wazuh-indexer.x86_64 0:4.11.0-1 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

===================================================================================================================================================================================================================
 Package                                                 Arch                                             Version                                            Repository                                       Size
===================================================================================================================================================================================================================
Updating:
 wazuh-indexer                                           x86_64                                           4.11.0-1                                           wazuh                                           831 M

Transaction Summary
===================================================================================================================================================================================================================
Upgrade  1 Package

Total download size: 831 M
Is this ok [y/d/N]: y
Downloading packages:
wazuh-indexer-4.11.0-1.x86_64.rpm                                                                                                                                                           | 831 MB  00:00:41     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Stop existing wazuh-indexer.service
touch: cannot touch ‘/var/lib/wazuh-indexer/tmp/wazuh-indexer.restart’: No such file or directory
error: %pre(wazuh-indexer-4.11.0-1.x86_64) scriptlet failed, exit status 1
Error in PREIN scriptlet in rpm package wazuh-indexer-4.11.0-1.x86_64
wazuh-indexer-4.10.1-1.x86_64 was supposed to be removed but is not!
  Verifying  : wazuh-indexer-4.10.1-1.x86_64                                                                                                                                                                   1/2 
  Verifying  : wazuh-indexer-4.11.0-1.x86_64                                                                                                                                                                   2/2 

Failed:
  wazuh-indexer.x86_64 0:4.10.1-1                                                                          wazuh-indexer.x86_64 0:4.11.0-1                                                                         

Complete!
Debian package is not affected. Upgrade 4.10.1 to 4.11.0 without stopping the service 🟢
root@ubuntu22:/home/vagrant# apt-get -y install wazuh-indexer=4.10.1-1
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  wazuh-indexer
0 upgraded, 1 newly installed, 0 to remove and 193 not upgraded.
Need to get 870 MB of archives.
After this operation, 1,097 MB of additional disk space will be used.
Get:1 https://packages.wazuh.com/4.x/apt stable/main amd64 wazuh-indexer amd64 4.10.1-1 [870 MB]
Fetched 870 MB in 10s (85.6 MB/s)                                                                                                                                                                                 
Selecting previously unselected package wazuh-indexer.
(Reading database ... 76328 files and directories currently installed.)
Preparing to unpack .../wazuh-indexer_4.10.1-1_amd64.deb ...
Running Wazuh Indexer Pre-Installation Script
Unpacking wazuh-indexer (4.10.1-1) ...
Setting up wazuh-indexer (4.10.1-1) ...
Running Wazuh Indexer Post-Installation Script
### NOT starting on installation, please execute the following statements to configure wazuh-indexer service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable wazuh-indexer.service
### You can start wazuh-indexer service by executing
 sudo systemctl start wazuh-indexer.service
Scanning processes...                                                                                                                                                                                              
Scanning linux images...                                                                                                                                                                                           

Running kernel seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.
root@ubuntu22:/home/vagrant# NODE_NAME=node-1
root@ubuntu22:/home/vagrant# mkdir /etc/wazuh-indexer/certs
tar -xf ./wazuh-certificates.tar -C /etc/wazuh-indexer/certs/ ./$NODE_NAME.pem ./$NODE_NAME-key.pem ./admin.pem ./admin-key.pem ./root-ca.pem
mv -n /etc/wazuh-indexer/certs/$NODE_NAME.pem /etc/wazuh-indexer/certs/indexer.pem
mv -n /etc/wazuh-indexer/certs/$NODE_NAME-key.pem /etc/wazuh-indexer/certs/indexer-key.pem
chmod 500 /etc/wazuh-indexer/certs
chmod 400 /etc/wazuh-indexer/certs/*
chown -R wazuh-indexer:wazuh-indexer /etc/wazuh-indexer/certs
root@ubuntu22:/home/vagrant# systemctl start wazuh-indexer
root@ubuntu22:/home/vagrant# systemctl status wazuh-indexer
● wazuh-indexer.service - wazuh-indexer
     Loaded: loaded (/lib/systemd/system/wazuh-indexer.service; disabled; vendor preset: enabled)
     Active: active (running) since Fri 2025-02-28 00:27:19 UTC; 9s ago
       Docs: https://documentation.wazuh.com
   Main PID: 3557 (java)
      Tasks: 66 (limit: 9389)
     Memory: 1.3G
        CPU: 21.483s
     CGroup: /system.slice/wazuh-indexer.service
             └─3557 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=t>

Feb 28 00:27:12 ubuntu22 systemd-entrypoint[3557]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.16.0.jar)
Feb 28 00:27:12 ubuntu22 systemd-entrypoint[3557]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Feb 28 00:27:12 ubuntu22 systemd-entrypoint[3557]: WARNING: System::setSecurityManager will be removed in a future release
Feb 28 00:27:13 ubuntu22 systemd-entrypoint[3557]: Feb 28, 2025 12:27:13 AM sun.util.locale.provider.LocaleProviderAdapter <clinit>
Feb 28 00:27:13 ubuntu22 systemd-entrypoint[3557]: WARNING: COMPAT locale provider will be removed in a future release
Feb 28 00:27:13 ubuntu22 systemd-entrypoint[3557]: WARNING: A terminally deprecated method in java.lang.System has been called
Feb 28 00:27:13 ubuntu22 systemd-entrypoint[3557]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.16.0.jar)
Feb 28 00:27:13 ubuntu22 systemd-entrypoint[3557]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Feb 28 00:27:13 ubuntu22 systemd-entrypoint[3557]: WARNING: System::setSecurityManager will be removed in a future release
Feb 28 00:27:19 ubuntu22 systemd[1]: Started wazuh-indexer.
root@ubuntu22:/home/vagrant# apt-get install wazuh-indexer=4.11.0-1
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be upgraded:
  wazuh-indexer
1 upgraded, 0 newly installed, 0 to remove and 193 not upgraded.
Need to get 870 MB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 https://packages.wazuh.com/4.x/apt stable/main amd64 wazuh-indexer amd64 4.11.0-1 [870 MB]
Fetched 870 MB in 11s (77.9 MB/s)                                                                                                                                                                                 
(Reading database ... 77509 files and directories currently installed.)
Preparing to unpack .../wazuh-indexer_4.11.0-1_amd64.deb ...
Running Wazuh Indexer Pre-Installation Script
Stop existing wazuh-indexer.service
Unpacking wazuh-indexer (4.11.0-1) over (4.10.1-1) ...
Setting up wazuh-indexer (4.11.0-1) ...
Running Wazuh Indexer Post-Installation Script
Restarting wazuh-indexer service...
Scanning processes...                                                                                                                                                                                              
Scanning linux images...                                                                                                                                                                                           

Running kernel seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.
root@ubuntu22:/home/vagrant# systemctl status wazuh-indexer
● wazuh-indexer.service - wazuh-indexer
     Loaded: loaded (/lib/systemd/system/wazuh-indexer.service; disabled; vendor preset: enabled)
     Active: active (running) since Fri 2025-02-28 00:28:25 UTC; 8s ago
       Docs: https://documentation.wazuh.com
   Main PID: 5053 (java)
      Tasks: 72 (limit: 9389)
     Memory: 1.3G
        CPU: 19.897s
     CGroup: /system.slice/wazuh-indexer.service
             └─5053 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=t>

Feb 28 00:28:19 ubuntu22 systemd-entrypoint[5053]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.16.0.jar)
Feb 28 00:28:19 ubuntu22 systemd-entrypoint[5053]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Feb 28 00:28:19 ubuntu22 systemd-entrypoint[5053]: WARNING: System::setSecurityManager will be removed in a future release
Feb 28 00:28:20 ubuntu22 systemd-entrypoint[5053]: Feb 28, 2025 12:28:20 AM sun.util.locale.provider.LocaleProviderAdapter <clinit>
Feb 28 00:28:20 ubuntu22 systemd-entrypoint[5053]: WARNING: COMPAT locale provider will be removed in a future release
Feb 28 00:28:20 ubuntu22 systemd-entrypoint[5053]: WARNING: A terminally deprecated method in java.lang.System has been called
Feb 28 00:28:20 ubuntu22 systemd-entrypoint[5053]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.16.0.jar)
Feb 28 00:28:20 ubuntu22 systemd-entrypoint[5053]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Feb 28 00:28:20 ubuntu22 systemd-entrypoint[5053]: WARNING: System::setSecurityManager will be removed in a future release
Feb 28 00:28:25 ubuntu22 systemd[1]: Started wazuh-indexer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
level/task Task issue type/bug Bug issue
Projects
Status: On hold
2 participants