[Bug]: Cannot release opensearch-ruby 3.2.0 because of Jenkins #4624

nhtruong · 2024-04-10T20:49:12Z

Describe the bug

The release of Ruby 3.2.0 was first triggered in Feb 2024 and failed with the following error:

/var/jenkins/workspace/opensearch-ruby-gems-release@tmp/durable-aa0447d1/script.sh: line 3: /usr/share/opensearch/.rvm/scripts/rvm: No such file or directory
ERROR:  Error installing opensearch-ruby-3.2.0.gem:
	The last version of faraday-net_http (>= 2.0, < 3.2) to support your Ruby & RubyGems was 3.0.2. Try installing it with `gem install faraday-net_http -v 3.0.2` and then running the current command again
	faraday-net_http requires Ruby version >= 3.0.0. The current ruby version is 2.6.0.0.
Successfully installed multi_json-1.15.0
Successfully installed uri-0.13.0
Successfully installed net-http-0.4.1
Gem 'opensearch-ruby' is not installed
WARNING:  faraday-net_http-3.1.0 is not signed
ERROR:  Error installing opensearch-ruby-3.2.0.gem:
	The last version of faraday-net_http (>= 2.0, < 3.2) to support your Ruby & RubyGems was 3.0.2. Try installing it with `gem install faraday-net_http -v 3.0.2` and then running the current command again
	faraday-net_http requires Ruby version >= 3.0.0. The current ruby version is 2.6.0.0.
Gem 'opensearch-ruby' is not installed

We tried to upgrade the rubyVersion of Jenkins to 3.0.6 but Jenkins still failed with the exact error.

We notice in the error message that RVM could not pick up the correct Ruby version:

/var/jenkins/workspace/opensearch-ruby-gems-release@tmp/durable-aa0447d1/script.sh: line 3: /usr/share/opensearch/.rvm/scripts/rvm: No such file or director

To reproduce

Build and install the gem with Ruby 2.6

opensearch-project/opensearch-ruby#238 (comment)

Additional context

Check out the discussion on the Ruby client repo.

The text was updated successfully, but these errors were encountered:

gaiksaya · 2024-04-10T21:49:40Z

The docker image used in the release: opensearchstaging/ci-runner:release-centos7-clients-v4 (see code)

I tried pulling the image and looking at the installed ruby versions.
This is the list:

[root@5df978f832bc opensearch]# rvm list
Warning! PATH is not properly set up, /usr/share/opensearch/.gem/bin is not available.
         Usually this is caused by shell initialization files. Search for PATH=... entries.
         You can also re-add RVM to your profile by running: rvm get stable --auto-dotfiles
         To fix it temporarily in this shell session run: rvm use .gem
         To ignore this error add rvm_silence_path_mismatch_check_flag=1 to your ~/.rvmrc file.
   jruby-9.3.0.0 [ x86_64 ]
 * ruby-2.6.0 [ x86_64 ]

# => - current
# =* - current && default
#  * - default

Looks like these are only versions installed on the docker file: https://github.com/opensearch-project/opensearch-build/blob/main/docker/ci/dockerfiles/current/release.centos7.clients.x64.arm64.dockerfile#L138-L141

@peterzhuamazon I believe for additional versions we either need to update the image or have something generic that would install it on the go?

Earlopain · 2024-04-11T08:44:09Z

That seems to be the way to go, yes. Can you try running /usr/share/opensearch/.rvm/scripts/rvm instead? That is the command actually being executed at the moment. I believe this path wasn't updated when 01613af was done.

So there needs to be made a change at https://github.com/opensearch-project/opensearch-build-libraries/blob/6d35e942e98e1acddd88633a2aa1bbe39ab5e002/vars/publishToRubyGems.groovy#L27 as well. Do these things actually run as root? My first instict is that they would run as whatever CONTAINER_USER is set to, so the command could just be ~/.rvm/scripts/rvm I think.

https://build.ci.opensearch.org/blue/organizations/jenkins/opensearch-ruby-gems-release/detail/opensearch-ruby-gems-release/29/pipeline/ which was run before that commit about changing the user doesn't show the rvm error. I've openend opensearch-project/opensearch-build-libraries#410

So, two issues. A more recent ruby version needs to be added to the image anyways.

gaiksaya · 2024-04-11T19:13:58Z

This is what I see when I use above command:

[opensearch@0628d16dc11d ~]$ rvm list
Warning! PATH is not properly set up, /usr/share/opensearch/.gem/bin is not available.
         Usually this is caused by shell initialization files. Search for PATH=... entries.
         You can also re-add RVM to your profile by running: rvm get stable --auto-dotfiles
         To fix it temporarily in this shell session run: rvm use .gem
         To ignore this error add rvm_silence_path_mismatch_check_flag=1 to your ~/.rvmrc file.
   jruby-9.3.0.0 [ x86_64 ]
 * ruby-2.6.0 [ x86_64 ]

# => - current
# =* - current && default
#  * - default

[opensearch@0628d16dc11d ~]$ /usr/share/opensearch/.rvm/scripts/rvm list
[opensearch@0628d16dc11d ~]$ which rvm
~/.rvm/bin/rvm

Note that I am logged in as opensearch user in the docker image.

What version of ruby would need to be installed?

Earlopain · 2024-04-11T19:22:33Z

The release action currently uses 3.1, mirroring that would be ideal I think. See https://github.com/opensearch-project/opensearch-ruby/blob/bfd43cf44775981d719ebfa00fb420560c0d2dac/.github/workflows/release_drafter.yml#L29-L32

The latest patch release at this time is 3.1.4.

peterzhuamazon · 2024-04-11T22:57:09Z

I would need to update the ruby from 2.6.0 to 3.0.2+, need to check the rvm support and rebuild images with changes.

peterzhuamazon · 2024-04-11T23:15:16Z


[ci-runner@88850ad06f30 ~]$ ruby --version
ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux]
[ci-runner@88850ad06f30 ~]$ rvm list
Warning! PATH is not properly set up, /home/ci-runner/.gem/bin is not available.
         Usually this is caused by shell initialization files. Search for PATH=... entries.
         You can also re-add RVM to your profile by running: rvm get stable --auto-dotfiles
         To fix it temporarily in this shell session run: rvm use .gem
         To ignore this error add rvm_silence_path_mismatch_check_flag=1 to your ~/.rvmrc file.
   jruby-9.3.0.0 [ x86_64 ]
 * ruby-3.1.2 [ x86_64 ]

# => - current
# =* - current && default
#  * - default

Send PR soon.

peterzhuamazon · 2024-04-11T23:17:26Z

Able to build:


[ci-runner@88850ad06f30 opensearch-ruby]$ gem build opensearch-ruby.gemspec
WARNING:  open-ended dependency on multi_json (>= 1.0) is not recommended
  if multi_json is semantically versioned, use:
    add_runtime_dependency 'multi_json', '~> 1.0'
WARNING:  See https://guides.rubygems.org/specification-reference/ for help
  Successfully built RubyGem
  Name: opensearch-ruby
  Version: 3.2.0
  File: opensearch-ruby-3.2.0.gem

peterzhuamazon · 2024-04-11T23:19:29Z

Hi @nhtruong please confirm if ruby version 3.1.2 is good for you.
I try to build opensearch-rb main branch with that and it seems fine.

Upgrade ruby from 2.6.0 to 3.1.2 for opensearch-ruby 3.2.0 requirements #4632

Thanks.

peterzhuamazon · 2024-04-12T20:18:24Z

The above PR merged and will have the image available in 5 hours or so due to the complexity of this image.

We can try to release next week.

Thanks.

peterzhuamazon · 2024-04-12T20:18:56Z

The opensearch-ruby workflow need to switch to 3.1.2 ruby before run the build.
@gaiksaya to apply the corresponding changes as it defaults to 2.6.0 still.

Thanks.

Earlopain · 2024-04-12T20:36:33Z

That sound good, thank you for your work. Is opensearch-project/opensearch-build-libraries#410 not an issue for the release?

Earlopain · 2024-04-14T19:45:46Z

I have openend PR opensearch-project/opensearch-build-libraries#413 for the above issue.

nhtruong · 2024-04-15T21:30:57Z

Attempted to redo the release but still failed. With different errors this time: https://build.ci.opensearch.org/blue/organizations/jenkins/opensearch-ruby-gems-release/detail/opensearch-ruby-gems-release/33/pipeline

gaiksaya · 2024-04-15T21:32:58Z

Hi @nhtruong
We are yet to make changes to your jenkinsFile. See attached PR above, that is yet to be merged and then another PR needs to be opened in ruby repo.
If this is urgent we can merge above PR and additional changes I can create another one.

nhtruong · 2024-04-16T16:47:23Z

We got past the previous error but got another one: https://build.ci.opensearch.org/blue/organizations/jenkins/opensearch-ruby-gems-release/detail/opensearch-ruby-gems-release/34/pipeline

ERROR:  While executing gem ... (Gem::Security::Exception)
    root cert /CN=opensearch/DC=amazon/DC=com is not trusted
Gem 'opensearch-ruby' is not installed

Earlopain · 2024-04-16T17:42:00Z

Maybe the gem cert --add in opensearch-build has to happen after switching versions? I'm not able to test myself at the moment but the fact that this happens so early stands out to me.

gaiksaya · 2024-04-16T18:25:16Z

I think you are right @Earlopain
I just tried reproducing the error. However forgot to use default ruby (2.6.0) to add cert. So basically did what you suggested:

docker run -u root -it opensearchstaging/ci-runner:release-centos7-clients-v4 /bin/bash
[root@e314dcfc5515 ci-runner]# git clone https://github.com/opensearch-project/opensearch-ruby.git
Cloning into 'opensearch-ruby'...
remote: Enumerating objects: 29553, done.
remote: Counting objects: 100% (932/932), done.
remote: Compressing objects: 100% (364/364), done.
remote: Total 29553 (delta 628), reused 793 (delta 553), pack-reused 28621
Receiving objects: 100% (29553/29553), 6.66 MiB | 3.08 MiB/s, done.
Resolving deltas: 100% (20467/20467), done.
[root@e314dcfc5515 ci-runner]# cd opensearch-ruby/
[root@e314dcfc5515 opensearch-ruby]# source /home/ci-runner/.rvm/scripts/rvm
[root@e314dcfc5515 opensearch-ruby]# rvm use 3.1.2
Using /home/ci-runner/.rvm/gems/ruby-3.1.2
[root@e314dcfc5515 opensearch-ruby]# ruby --version
ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux]
[root@e314dcfc5515 opensearch-ruby]# gem cert --add .github/opensearch-rubygems.pem
Added '/CN=opensearch/DC=amazon/DC=com'
[root@e314dcfc5515 opensearch-ruby]# curl -J -L -H 'Accept: application/octet-stream' -H 'Authorization: Bearer <token>' https://api.github.com/repos/opensearch-project/opensearch-ruby/releases/assets/162445549 -o artifacts.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  140k  100  140k    0     0   109k      0  0:00:01  0:00:01 --:--:-- 3783k
[root@e314dcfc5515 opensearch-ruby]# ls
ADMINS.md     CODE_OF_CONDUCT.md  CONTRIBUTING.md     Gemfile      MAINTAINERS.md  NOTICE.txt      README.md     Rakefile     UPGRADING.md   api_generator     bin     jenkins  opensearch-ruby.gemspec  samples  test
CHANGELOG.md  COMPATIBILITY.md    DEVELOPER_GUIDE.md  LICENSE.txt  Makefile        OpenSearch.svg  RELEASING.md  SECURITY.md  USER_GUIDE.md  artifacts.tar.gz  guides  lib      profile                  spec
[root@e314dcfc5515 opensearch-ruby]# tar -xvf artifacts.tar.gz
dist/
dist/opensearch-ruby-3.2.0.gem
[root@e314dcfc5515 opensearch-ruby]# cd dist/
[root@e314dcfc5515 dist]# gemNameWithVersion=$(ls *.gem)
[root@e314dcfc5515 dist]# echo $gemNameWithVersion
opensearch-ruby-3.2.0.gem
[root@e314dcfc5515 dist]# gem install $gemNameWithVersion
Fetching faraday-2.9.0.gem
Fetching multi_json-1.15.0.gem
Fetching faraday-net_http-3.1.0.gem
Successfully installed multi_json-1.15.0
Successfully installed faraday-net_http-3.1.0
Successfully installed faraday-2.9.0
Successfully installed opensearch-ruby-3.2.0
Parsing documentation for multi_json-1.15.0
Installing ri documentation for multi_json-1.15.0
Parsing documentation for faraday-net_http-3.1.0
Installing ri documentation for faraday-net_http-3.1.0
Parsing documentation for faraday-2.9.0
Installing ri documentation for faraday-2.9.0
Parsing documentation for opensearch-ruby-3.2.0
Installing ri documentation for opensearch-ruby-3.2.0
Done installing documentation for multi_json, faraday-net_http, faraday, opensearch-ruby after 26 seconds
4 gems installed
[root@e314dcfc5515 dist]# gemName=$(echo $gemNameWithVersion | sed -E 's/(-[0-9.]+-*[a-z]*.gem$)/g')
sed: -e expression #1, char 27: unterminated `s' command
[root@e314dcfc5515 dist]# gemName=$(echo $gemNameWithVersion | sed -E 's/(-[0-9.]+-*[a-z]*.gem$)//g')
[root@e314dcfc5515 dist]# echo $gemName
opensearch-ruby
[root@e314dcfc5515 dist]# gem uninstall $gemName
Remove executables:
	opensearch_ruby_console

in addition to the gem? [Yn]  Y
Removing opensearch_ruby_console
Successfully uninstalled opensearch-ruby-3.2.0
[root@e314dcfc5515 dist]# gem install $gemNameWithVersion -P MediumSecurity
Successfully installed opensearch-ruby-3.2.0
Parsing documentation for opensearch-ruby-3.2.0
Installing ri documentation for opensearch-ruby-3.2.0
Done installing documentation for opensearch-ruby after 23 seconds
1 gem installed
[root@e314dcfc5515 dist]# gem uninstall $gemName
Remove executables:
	opensearch_ruby_console

in addition to the gem? [Yn]  Y
Removing opensearch_ruby_console
Successfully uninstalled opensearch-ruby-3.2.0
[root@e314dcfc5515 dist]# gem install $gemNameWithVersion -P HighSecurity
Successfully installed opensearch-ruby-3.2.0
Parsing documentation for opensearch-ruby-3.2.0
Installing ri documentation for opensearch-ruby-3.2.0
Done installing documentation for opensearch-ruby after 32 seconds
1 gem installed

Will check if exact steps in the code to see if that is the issue.

gaiksaya · 2024-04-16T18:36:32Z

And our theory is right. Ruby version needs to be switched before the cert is added. Did not think that would make a difference

docker run -u root -it opensearchstaging/ci-runner:release-centos7-clients-v4 /bin/bash
[root@9d94be570f33 ci-runner]# git clone https://github.com/opensearch-project/opensearch-ruby.git
Cloning into 'opensearch-ruby'...
remote: Enumerating objects: 29553, done.
remote: Counting objects: 100% (932/932), done.
remote: Compressing objects: 100% (364/364), done.
remote: Total 29553 (delta 628), reused 793 (delta 553), pack-reused 28621
Receiving objects: 100% (29553/29553), 6.66 MiB | 5.72 MiB/s, done.
Resolving deltas: 100% (20467/20467), done.
[root@9d94be570f33 ci-runner]# cd opensearch-ruby/
[root@9d94be570f33 opensearch-ruby]# gem cert --add .github/opensearch-rubygems.pem
Added '/CN=opensearch/DC=amazon/DC=com'
[root@9d94be570f33 opensearch-ruby]# source /home/ci-runner/.rvm/scripts/rvm
[root@9d94be570f33 opensearch-ruby]# rvm use 3.1.2
Using /home/ci-runner/.rvm/gems/ruby-3.1.2
[root@9d94be570f33 opensearch-ruby]# ruby --version
ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux]
[root@9d94be570f33 opensearch-ruby]# curl -J -L -H 'Accept: application/octet-stream' -H 'Authorization: Bearer <token>' https://api.github.com/repos/opensearch-project/opensearch-ruby/releases/assets/162445549 -o artifacts.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  140k  100  140k    0     0   131k      0  0:00:01  0:00:01 --:--:--  131k
[root@9d94be570f33 opensearch-ruby]# tar -xvf artifacts.tar.gz
dist/
dist/opensearch-ruby-3.2.0.gem
[root@9d94be570f33 opensearch-ruby]# cd dist/
[root@9d94be570f33 dist]# gemNameWithVersion=$(ls *.gem)
[root@9d94be570f33 dist]# gem install $gemNameWithVersion
Fetching faraday-2.9.0.gem
Fetching faraday-net_http-3.1.0.gem
Fetching multi_json-1.15.0.gem
Successfully installed multi_json-1.15.0
Successfully installed faraday-net_http-3.1.0
Successfully installed faraday-2.9.0
Successfully installed opensearch-ruby-3.2.0
Parsing documentation for multi_json-1.15.0
Installing ri documentation for multi_json-1.15.0
Parsing documentation for faraday-net_http-3.1.0
Installing ri documentation for faraday-net_http-3.1.0
Parsing documentation for faraday-2.9.0
Installing ri documentation for faraday-2.9.0
Parsing documentation for opensearch-ruby-3.2.0
Installing ri documentation for opensearch-ruby-3.2.0
Done installing documentation for multi_json, faraday-net_http, faraday, opensearch-ruby after 22 seconds
4 gems installed
[root@9d94be570f33 dist]# gemName=$(echo $gemNameWithVersion | sed -E 's/(-[0-9.]+-*[a-z]*.gem$)/g')
sed: -e expression #1, char 27: unterminated `s' command
[root@9d94be570f33 dist]# gemName=$(echo $gemNameWithVersion | sed -E 's/(-[0-9.]+-*[a-z]*.gem$)//g')
[root@9d94be570f33 dist]# gem uninstall $gemName
Remove executables:
	opensearch_ruby_console

in addition to the gem? [Yn]  Y
Removing opensearch_ruby_console
Successfully uninstalled opensearch-ruby-3.2.0
[root@9d94be570f33 dist]# gem install $gemNameWithVersion -P MediumSecurity
ERROR:  While executing gem ... (Gem::Security::Exception)
    root cert /CN=opensearch/DC=amazon/DC=com is not trusted

Will create a PR to fix this.

peterzhuamazon · 2024-04-16T22:01:03Z

It is released: https://build.ci.opensearch.org/job/opensearch-ruby-gems-release/35/console

gaiksaya · 2024-04-16T22:04:17Z

Closing it as the issue is resolved now!

nhtruong added bug Something isn't working untriaged Issues that have not yet been triaged labels Apr 10, 2024

nhtruong mentioned this issue Apr 10, 2024

Build and install the gem during the release workflow opensearch-project/opensearch-ruby#238

Closed

gaiksaya removed the untriaged Issues that have not yet been triaged label Apr 10, 2024

Earlopain mentioned this issue Apr 11, 2024

[Bug]: Cannot release opensearch-ruby 3.2.0 because of publishToRubyGems RVM issues opensearch-project/opensearch-build-libraries#410

Closed

peterzhuamazon mentioned this issue Apr 11, 2024

Upgrade ruby from 2.6.0 to 3.1.2 for opensearch-ruby 3.2.0 requirements #4632

Merged

gaiksaya mentioned this issue Apr 15, 2024

Switch ruby version to 3.1.2 on release pipeline opensearch-project/opensearch-ruby#241

Merged

nhtruong closed this as completed in opensearch-project/opensearch-ruby#241 Apr 15, 2024

nhtruong reopened this Apr 16, 2024

github-actions bot added the untriaged Issues that have not yet been triaged label Apr 16, 2024

gaiksaya removed the untriaged Issues that have not yet been triaged label Apr 16, 2024

gaiksaya self-assigned this Apr 16, 2024

peterzhuamazon self-assigned this Apr 16, 2024

gaiksaya mentioned this issue Apr 16, 2024

Fix setting ruby version opensearch-project/opensearch-build-libraries#417

Merged

gaiksaya closed this as completed Apr 16, 2024

peterzhuamazon added this to OpenSearch Engineering Effectiveness Apr 16, 2024

peterzhuamazon moved this to In Progress in OpenSearch Engineering Effectiveness Apr 16, 2024

peterzhuamazon moved this from In Progress to Done in OpenSearch Engineering Effectiveness Apr 16, 2024

peterzhuamazon added the jenkins Jenkins related issue label Apr 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: Cannot release opensearch-ruby 3.2.0 because of Jenkins #4624

[Bug]: Cannot release opensearch-ruby 3.2.0 because of Jenkins #4624

nhtruong commented Apr 10, 2024 •

edited

Loading

gaiksaya commented Apr 10, 2024 •

edited

Loading

Earlopain commented Apr 11, 2024 •

edited

Loading

gaiksaya commented Apr 11, 2024

Earlopain commented Apr 11, 2024

peterzhuamazon commented Apr 11, 2024 •

edited

Loading

peterzhuamazon commented Apr 11, 2024

peterzhuamazon commented Apr 11, 2024 •

edited

Loading

peterzhuamazon commented Apr 11, 2024 •

edited

Loading

peterzhuamazon commented Apr 12, 2024

peterzhuamazon commented Apr 12, 2024

Earlopain commented Apr 12, 2024

Earlopain commented Apr 14, 2024

nhtruong commented Apr 15, 2024

gaiksaya commented Apr 15, 2024

nhtruong commented Apr 16, 2024

Earlopain commented Apr 16, 2024 •

edited

Loading

gaiksaya commented Apr 16, 2024 •

edited

Loading

gaiksaya commented Apr 16, 2024 •

edited

Loading

peterzhuamazon commented Apr 16, 2024 •

edited

Loading

gaiksaya commented Apr 16, 2024

[Bug]: Cannot release opensearch-ruby 3.2.0 because of Jenkins #4624

[Bug]: Cannot release opensearch-ruby 3.2.0 because of Jenkins #4624

Comments

nhtruong commented Apr 10, 2024 • edited Loading

Describe the bug

To reproduce

Additional context

gaiksaya commented Apr 10, 2024 • edited Loading

Earlopain commented Apr 11, 2024 • edited Loading

gaiksaya commented Apr 11, 2024

Earlopain commented Apr 11, 2024

peterzhuamazon commented Apr 11, 2024 • edited Loading

peterzhuamazon commented Apr 11, 2024

peterzhuamazon commented Apr 11, 2024 • edited Loading

peterzhuamazon commented Apr 11, 2024 • edited Loading

peterzhuamazon commented Apr 12, 2024

peterzhuamazon commented Apr 12, 2024

Earlopain commented Apr 12, 2024

Earlopain commented Apr 14, 2024

nhtruong commented Apr 15, 2024

gaiksaya commented Apr 15, 2024

nhtruong commented Apr 16, 2024

Earlopain commented Apr 16, 2024 • edited Loading

gaiksaya commented Apr 16, 2024 • edited Loading

gaiksaya commented Apr 16, 2024 • edited Loading

peterzhuamazon commented Apr 16, 2024 • edited Loading

gaiksaya commented Apr 16, 2024

nhtruong commented Apr 10, 2024 •

edited

Loading

gaiksaya commented Apr 10, 2024 •

edited

Loading

Earlopain commented Apr 11, 2024 •

edited

Loading

peterzhuamazon commented Apr 11, 2024 •

edited

Loading

peterzhuamazon commented Apr 11, 2024 •

edited

Loading

peterzhuamazon commented Apr 11, 2024 •

edited

Loading

Earlopain commented Apr 16, 2024 •

edited

Loading

gaiksaya commented Apr 16, 2024 •

edited

Loading

gaiksaya commented Apr 16, 2024 •

edited

Loading

peterzhuamazon commented Apr 16, 2024 •

edited

Loading