[Bug]: Logs Not available in test-report

[derek-ho]

Describe the bug

Coming from @peternied , logs from failed autocuts do not contain actual links or logs, instead showing as "Not Available"

To reproduce

Download test report from autocut, example: https://ci.opensearch.org/ci/dbc/integ-test/2.12.0/9167/linux/x64/tar/test-results/6950/integ-test/test-report.yml](https://ci.opensearch.org/ci/dbc/integ-test/2.12.0/9167/linux/x64/tar/test-results/6950/integ-test/test-report.yml), you will see "Not available", instead of a link to the logs or failures

Expected behavior

No response

Screenshots

If applicable, add screenshots to help explain your problem.

Host / Environment

No response

Additional context

No response

Relevant log output

No response

[gaiksaya]

Hi @derek-ho ,

This might be helpful: https://github.com/opensearch-project/opensearch-build/tree/main/src/report_workflow#why-are-some-component-testing-results-missing-or-unavailable

@zelinh Can add more details.

Thanks!

[rishabh6788]

The reason for status showing N/A is that OpenSearch process failed to come up. We are now uploading the OpenSearch logs to S3 bucket for users to be able to download and see why OS process failed to come up. For the e.g. below is the log for test status not showing:

Exception in thread "main" SettingsException[Failed to load settings from [opensearch.yml]]; nested: JsonParseException[Duplicate field 'node.name'
 at [Source: (sun.nio.ch.ChannelInputStream); line: 166, column: 10]];
	at org.opensearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1146)
	at org.opensearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1117)
	at org.opensearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:96)
	at org.opensearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:118)
	at org.opensearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:109)
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
	at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
	at org.opensearch.cli.Command.main(Command.java:101)
	at org.opensearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:56)
Caused by: com.fasterxml.jackson.core.JsonParseException: Duplicate field 'node.name'
 at [Source: (sun.nio.ch.ChannelInputStream); line: 166, column: 10]
	at com.fasterxml.jackson.core.json.JsonReadContext._checkDup(JsonReadContext.java:250)
	at com.fasterxml.jackson.core.json.JsonReadContext.setCurrentName(JsonReadContext.java:244)
	at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:482)
	at org.opensearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:66)
	at org.opensearch.common.settings.Settings.fromXContent(Settings.java:660)
	at org.opensearch.common.settings.Settings.fromXContent(Settings.java:629)
	at org.opensearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1142)
	... 10 more
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/tmp/tmpx9mwdyf7/1/local-test-cluster/opensearch-2.12.0/lib/opensearch-2.12.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
Exception in thread "main" SettingsException[Failed to load settings from [opensearch.yml]]; nested: JsonParseException[Duplicate field 'node.name'
 at [Source: (sun.nio.ch.ChannelInputStream); line: 166, column: 10]];
	at org.opensearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1146)
	at org.opensearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1117)
	at org.opensearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:96)
	at org.opensearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:118)
	at org.opensearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:109)
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
	at org.opensearch.cli.Command.main(Command.java:101)
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
Caused by: com.fasterxml.jackson.core.JsonParseException: Duplicate field 'node.name'
 at [Source: (sun.nio.ch.ChannelInputStream); line: 166, column: 10]
	at com.fasterxml.jackson.core.json.JsonReadContext._checkDup(JsonReadContext.java:250)
	at com.fasterxml.jackson.core.json.JsonReadContext.setCurrentName(JsonReadContext.java:244)
	at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:482)
	at org.opensearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:66)
	at org.opensearch.common.settings.Settings.fromXContent(Settings.java:660)
	at org.opensearch.common.settings.Settings.fromXContent(Settings.java:629)
	at org.opensearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1142)
	... 9 more

The error logs can be downloaded from https://ci.opensearch.org/ci/dbc/integ-test/2.12.0/9167/linux/x64/tar/test-results/6950/integ-test/security/with-security/local-cluster-logs/id-0/stderr.txt

@gaiksaya Can we add this link in the documentation for users to be able to see the cluster logs.

[peternied]

@rishabh6788 Can we fix the manifest to show those files even if the server failed to come up? Its difficult to infer the paths and associated files that are available.

[rishabh6788]

@rishabh6788 Can we fix the manifest to show those files even if the server failed to come up? Its difficult to infer the paths and associated files that are available.

It should be doable. Will look into it further.

[rishabh6788]

The PR to add local cluster logs to test report has been merged. You should be able to see the stdout and stderr log files in the newly generated test-report.yml files.
Closing this issue, feel free to reopen if you have any other concerns.
@peternied @derek-ho