-
Notifications
You must be signed in to change notification settings - Fork 283
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Debian(RPM later): Restart OpenSearch/Dashboards Service after Upgrading the Package #3891
Comments
[Untriage] |
I can confirm the problem and our Graylog test server was non functional for 3 weeks. |
it happend again... |
Hi @slalomsk8er , That is expected behavior as our service file does not enable opensearch by default. In our design we do not start opensearch service by default as we want users to control the runs of the software. I think you can write in your upgrade script to restart the service after upgrade if that is suitable. Thanks. |
Hi @peterzhuamazon, This is very unfortunate as this is the only service known to me, that isn't restarted after update on a Debian system. What is the reasoning to have a service and want to have the users to control the runs of the software? To me those are mostly mutually exclusive. I will consult with my integrator about this issue but I guess, if it isn't save enough for the opensearch-project to automatically restart the service, we will switch to manual updates for this package. Thanks for any clarification. |
Hi @slalomsk8er, Thanks for the responses. The initial design was based on our initial thoughts, and a restart on update complete, as you mentioned, is surely convenient to the end users. Since this is an open source project, we welcome community feedbacks and improvements. We welcome you to send a possible PR if you are able to suggest a fix to this. Or I can take a look on this when I have some time / bandwidth. Let me know, Thanks. |
I ran out of time but found some nice diagrams that could help MaintainerScripts. |
To my understanding if I put the trigger in postinst, then both install and upgrade will be affected unless I have conditions around them. This would make the script way too complicated to maintain. I am trying to find out if there is any systemd service related changes so if a service is already started, then an upgrade will just auto trigger the restart. |
I think I misremembered and Debian doesn't auto restart services. If you insist in stopping the service I think the post install script receives a argument that lets you know if it's a install or update. |
It is more like the postrm script. I add a service stop there so it will not have cases where user uninstall the pkg but the service still running with a zombie process in bg. |
I do, however, can try to see if we can add a condition to understand if it is a install/upgrade. But that might be different case between deb and rpm. If I have to change deb I have to change rpm as well to keep the consistency. Will check. |
https://www.debian.org/doc/manuals/debian-faq/pkg-basics.en.html#maintscripts
Maybe I wasn't hallucinating self restarting services after all. https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html
So maybe the |
Not starting the service on installation makes sense because on first start OS would bootstrap some config and data before the end-user had the opportunity to tune the system (e.g. change default insecure passwords). While these changes can be made later, this is a rather painful process, so violating the Debian policy of starting services on installation makes sense in some cases like this one. I believe this behavior should be kept. But if the OpenSearch service is running, it should be restarted during the update to minimize downtime. A similar behavior of not starting on install but restarting on upgrade is implemented in the following non-debian-provided packages I install on my Debian nodes for configuration management: Puppet & Choria: both need some "seeding" before they can start (the puppet agent needs to know which server it should talk to, and Choria must be configured via Puppet). The following PR implemented such a behavior for Choria in order to mimic the Puppet one: choria-io/go-choria#1392. I believe we can adopt a similar behavior. This can be part of the current WIP to improve OS / OSD packaging. |
More comments in: |
Re-open for RPM. |
After testing, this method does not seem to scale well with RPM due to RPM is actually verifying the old installation after new upgrades is already installed. Several times the postin restart of the new pkg would be interrupted by the preun stop from the old pkg, such as:
Note that the 2.11.1 stop happened actually after 2.13.0 start. Will close this issue for now as rpm is not processing this as stable as deb. Thanks. |
how can i disable the restart after upgrade? |
Hi @pitch-d , the condition that is checked for whether the service will be restarted after an upgraded is the "enabled" state of the SystemD-Unit (as returned by systemctl is-enabled opensearch.service). So as of now, I guess, a possibility would be to add tasks for disabling/re-enabling the service to the playbook to get the behaviour previous to OpenSearch 2.13.0 |
Describe the bug
I accept you to not consider this a bug at all and argue that this is desired behavior. I still like to bring this up for your consideration.
On upgrading from OpenSearch 2.7 to 2.8 the pre removal script stopped the OpenSearch service. However it is then not restarted automatically after the update.
To Reproduce
apt update
apt full-upgrage
to upgrade to OpenSearch 2.8 (or later I bet).Expected behavior
One can argue what the best behavior would be here.
What I see with upgrades of Elasticsearch Debian packages is that it does not stop the Elasticsearch service at all. This gives me an opportunity to reduce downtime. I usually get away with first doing the package upgrade while the Elasticsearch service is still running. Then I review configuration changes with
vimdiff
. And then I either restart the service or in case I have a kernel upgrade as well the whole system, probably after disabling shard allocation and/or pausing background work¹. One could consider this approach as being unclean or unsafe. However in practice it worked for Elasticsearch just fine for me.Another approach would be to still stop the OpenSearch service but then start it after the update in case the Systemd service is enabled.
I'd prefer the first approach to leave the service state alone during the upgrade. That leaves it up to the admin to decide how to approach the upgrade. Of course the documentation should have some best practice hints.
[1] See https://www.elastic.co/guide/en/elastic-stack/8.8/upgrading-elasticsearch.html
Plugins
Not relevant for this topic.
Screenshots
Excerpt of console output:
State of Systemd service after upgrade from OpenSearch 2.7 to 2.8:
Host/Environment (please complete the following information):
Additional context
Not required here I think.
The text was updated successfully, but these errors were encountered: