I/O: Split OutputManager creation from setup #3005

tcclevenger · 2024-09-18T22:28:14Z

Split initialize_output_managers() into two parts

create_output_managers(): Make output param list before atm processes or fields are created and construct OM from comm, param, time_stamp, is_restart.
initialize_output_managers(): Setup output managers with the field manager and grid manager. Remove comm, param, time_stamp, is_restart from OM::setup() (it was moved to constructor).

bartgol

Some thoughts on design. Feel free to push back if you disagree.

components/eamxx/src/control/atmosphere_driver.cpp

bartgol · 2024-09-18T22:55:02Z

components/eamxx/src/control/atmosphere_driver.cpp

+
+    // Check if the filename prefix for this file has already been set.  If not, use the simulation casename.
+    if (not params.isParameter("filename_prefix")) {
+      params.set<std::string>("filename_prefix",m_casename+".scream.h"+std::to_string(om_tally++));


I don't think we need the int after .h. Our naming convention is already taking care of differentiating two streams. If a user uses the same prefix for 2 streams that have the same output specs, well, then it's their fault. Though I suppose we could add a check in OutputManager, that ensures that when registering a file, we don't find out it was already registered...

components/eamxx/src/control/atmosphere_driver.cpp

E3SM-Bot · 2024-09-19T19:25:12Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 6061
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	I/O
PULLREQUESTNUM	3005
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`c189615`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`25120ff`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: tcclevenger

E3SM-Bot · 2024-09-19T19:33:26Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 6061
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	I/O
PULLREQUESTNUM	3005
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`c189615`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`25120ff`
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Weaver # 6061 FAILED (click to see last 100 lines of console output)


Warning: Permanently added the ECDSA host key for IP address '140.82.113.4' to the list of known hosts.
Submodule 'extern/Catch2' (git@github.com:E3SM-Project/Catch2) registered for path 'externals/ekat/extern/Catch2'
Submodule 'extern/kokkos' (git@github.com:E3SM-Project/kokkos) registered for path 'externals/ekat/extern/kokkos'
Submodule 'extern/spdlog' (git@github.com:gabime/spdlog.git) registered for path 'externals/ekat/extern/spdlog'
Submodule 'extern/yaml-cpp' (git@github.com:SNLComputation/yaml-cpp.git) registered for path 'externals/ekat/extern/yaml-cpp'
Cloning into '/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6061/scream/externals/ekat/extern/Catch2'...
Cloning into '/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6061/scream/externals/ekat/extern/kokkos'...
Warning: Permanently added the ECDSA host key for IP address '140.82.112.3' to the list of known hosts.
Cloning into '/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6061/scream/externals/ekat/extern/spdlog'...
Cloning into '/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6061/scream/externals/ekat/extern/yaml-cpp'...
ERROR: Repository not found.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights

and the repository exists.

fatal: clone of 'git@github.com:SNLComputation/yaml-cpp.git' into submodule path '/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6061/scream/externals/ekat/extern/yaml-cpp' failed

Failed to clone 'extern/yaml-cpp'. Retry scheduled

Cloning into '/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6061/scream/externals/ekat/extern/yaml-cpp'...

ERROR: Repository not found.

fatal: Could not read from remote repository.
Please make sure you have the correct access rights

and the repository exists.

fatal: clone of 'git@github.com:SNLComputation/yaml-cpp.git' into submodule path '/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6061/scream/externals/ekat/extern/yaml-cpp' failed

Failed to clone 'extern/yaml-cpp' a second time, aborting

Failed to recurse into submodule path 'externals/ekat'
at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2846)
at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2185)
at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.lambda$execute$0(CliGitAPIImpl.java:1573)
at Jenkins v2.462.1//com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at Jenkins v2.462.1//com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76)
at Jenkins v2.462.1//com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at Jenkins v2.462.1//com.google.common.util.concurrent.DirectExecutorService.execute(DirectExecutorService.java:51)
at java.base/java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:184)
at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.submitRemainingCommand(GitCommandsExecutor.java:77)
at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:70)

Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to weaver

at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1826)

at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)

at hudson.remoting.Channel.call(Channel.java:1042)

at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:153)

at jdk.internal.reflect.GeneratedMethodAccessor105.invoke(Unknown Source)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:569)

at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:138)

at PluginClassLoader for git-client/jdk.proxy30/jdk.proxy30.$Proxy100.execute(Unknown Source)

at PluginClassLoader for git//hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:196)

at PluginClassLoader for git//hudson.plugins.git.GitSCM.checkout(GitSCM.java:1388)

at hudson.scm.SCM.checkout(SCM.java:540)

at hudson.model.AbstractProject.checkout(AbstractProject.java:1247)

at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)

at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)

at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)

at hudson.model.Run.execute(Run.java:1894)

at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)

at hudson.model.ResourceController.execute(ResourceController.java:101)

at hudson.model.Executor.run(Executor.java:446)

Caused: hudson.plugins.git.GitException

at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.checkResult(GitCommandsExecutor.java:89)

at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:69)

at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:47)

at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.execute(CliGitAPIImpl.java:1576)

at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:170)

at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)

at hudson.remoting.UserRequest.perform(UserRequest.java:211)

at hudson.remoting.UserRequest.perform(UserRequest.java:54)

at hudson.remoting.Request$2.run(Request.java:377)

at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)

at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at java.base/java.lang.Thread.run(Thread.java:829)

Caused: java.io.IOException: Could not perform submodule update

at PluginClassLoader for git//hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:201)

at PluginClassLoader for git//hudson.plugins.git.GitSCM.checkout(GitSCM.java:1388)

at hudson.scm.SCM.checkout(SCM.java:540)

at hudson.model.AbstractProject.checkout(AbstractProject.java:1247)

at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)

at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)

at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)

at hudson.model.Run.execute(Run.java:1894)

at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)

at hudson.model.ResourceController.execute(ResourceController.java:101)

at hudson.model.Executor.run(Executor.java:446)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins7932543393656922168.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

E3SM-Bot · 2024-10-09T19:10:13Z

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

E3SM-Bot · 2024-10-09T19:13:07Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 6138
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	I/O;AT: RETEST
PULLREQUESTNUM	3005
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`7cc2511`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`c41c698`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5900
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	I/O;AT: RETEST
PULLREQUESTNUM	3005
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`7cc2511`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`c41c698`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: tcclevenger

E3SM-Bot · 2024-10-09T21:22:59Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 6138
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	I/O;AT: RETEST
PULLREQUESTNUM	3005
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`7cc2511`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`c41c698`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5900
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	I/O;AT: RETEST
PULLREQUESTNUM	3005
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`7cc2511`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`c41c698`
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Weaver # 6138 FAILED (click to see last 100 lines of console output)


FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6138/scream/components/eamxx/ctest-build/full_sp_debug
===============================================================================
Testing '\''7cc2511566bccc163be1da79300eaca67d030d94'\'' for test '\''release'\''
===============================================================================
RUN: taskset -c 104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155 sh -c '\''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6138/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6138/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6138/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6138/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.1/gcc/11.3.0/openmpi/4.1.6/5tv5psl -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.2/gcc/11.3.0/openmpi/4.1.6/pyuuqd3 -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.6/2s52shy -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='\''\'\'''\''ON'\''\'\'''\'' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release" '\''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6138/scream/components/eamxx/ctest-build/release
===============================================================================
Testing '\''7cc2511566bccc163be1da79300eaca67d030d94'\'' for test '\''full_debug'\''
===============================================================================
RUN: taskset -c 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51 sh -c '\''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6138/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6138/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6138/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6138/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.1/gcc/11.3.0/openmpi/4.1.6/5tv5psl -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.2/gcc/11.3.0/openmpi/4.1.6/pyuuqd3 -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.6/2s52shy -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DKokkos_ENABLE_DEBUG_BOUNDS_CHECK=True -DEKAT_DISABLE_TPL_WARNINGS='\''\'\'''\''ON'\''\'\'''\'' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '\''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6138/scream/components/eamxx/ctest-build/full_debug
Build type full_debug failed at testing time. Here'\''s a list of failed tests:
97:surface_coupling_np1
151:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp
Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

89:surface_coupling_np1
Build type release failed at testing time. Here'''s a list of failed tests:

96:surface_coupling_np1

150:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp
Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6138/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6138/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6138/scream/components/eamxx

weaver failed'

errors='Build type full_debug failed at testing time. Here'''s a list of failed tests:

97:surface_coupling_np1

151:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp

Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

89:surface_coupling_np1
Build type release failed at testing time. Here'''s a list of failed tests:

96:surface_coupling_np1

150:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp
Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6138/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6138/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6138/scream/components/eamxx

weaver failed'

SA_FAILURES_DETAILS+='Build type full_debug failed at testing time. Here'''s a list of failed tests:

97:surface_coupling_np1

151:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp

Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

89:surface_coupling_np1
Build type release failed at testing time. Here'''s a list of failed tests:

96:surface_coupling_np1

150:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp
Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6138/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6138/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6138/scream/components/eamxx

weaver failed'

[[ 1 == 0 ]]
[[ weaver == \m\a\p\p\y ]]
set +x

######################################################

FAILS DETECTED:

SCREAM STANDALONE TESTING FAILED!

Build type full_debug failed at testing time. Here's a list of failed tests:

97:surface_coupling_np1

151:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp

Build type full_sp_debug failed at testing time. Here's a list of failed tests:

89:surface_coupling_np1
Build type release failed at testing time. Here's a list of failed tests:

96:surface_coupling_np1

150:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp
Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6138/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6138/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6138/scream/components/eamxx

weaver failed

######################################################

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins12785339563556331918.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

SCREAM_PullRequest_Autotester_Mappy # 5900 FAILED (click to see last 100 lines of console output)


Build type debug_nopack_fpe failed at testing time. Here'''s a list of failed tests:

313:surface_coupling_np1

314:surface_coupling_np2

315:surface_coupling_np3

316:surface_coupling_np4

498:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np2_vs_np1

499:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np3_vs_np1

500:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np4_vs_np1
Build type release failed at testing time. Here'''s a list of failed tests:

324:surface_coupling_np1

325:surface_coupling_np2

326:surface_coupling_np3

327:surface_coupling_np4

522:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np2_vs_np1

523:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np3_vs_np1

524:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np4_vs_np1

525:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp
Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5900/scream/components/eamxx && source /projects/sems/modulefiles/utils/sems-modules-init.sh && module purge && module load sems-cmake/3.27.9 sems-git/2.42.0 sems-gcc/11.4.0 sems-openmpi-no-cuda/4.1.6 sems-netcdf-c/4.9.2 sems-netcdf-cxx/4.2 sems-netcdf-fortran/4.6.1 sems-parallel-netcdf/1.12.3 sems-openblas && export GATOR_INITIAL_MB=4000MB && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy

RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5900/scream/components/eamxx && source /projects/sems/modulefiles/utils/sems-modules-init.sh && module purge && module load sems-cmake/3.27.9 sems-git/2.42.0 sems-gcc/11.4.0 sems-openmpi-no-cuda/4.1.6 sems-netcdf-c/4.9.2 sems-netcdf-cxx/4.2 sems-netcdf-fortran/4.6.1 sems-parallel-netcdf/1.12.3 sems-openblas && export GATOR_INITIAL_MB=4000MB && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5900/scream/components/eamxx

mappy failed'

SA_FAILURES_DETAILS+='Build type full_debug failed at testing time. Here'''s a list of failed tests:

325:surface_coupling_np1

326:surface_coupling_np2

327:surface_coupling_np3

328:surface_coupling_np4

523:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np2_vs_np1

524:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np3_vs_np1

525:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np4_vs_np1

526:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp

Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

290:surface_coupling_np1

291:surface_coupling_np2

292:surface_coupling_np3

293:surface_coupling_np4
Build type debug_nopack_fpe failed at testing time. Here'''s a list of failed tests:

313:surface_coupling_np1

314:surface_coupling_np2

315:surface_coupling_np3

316:surface_coupling_np4

498:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np2_vs_np1

499:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np3_vs_np1

500:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np4_vs_np1
Build type release failed at testing time. Here'''s a list of failed tests:

324:surface_coupling_np1

325:surface_coupling_np2

326:surface_coupling_np3

327:surface_coupling_np4

522:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np2_vs_np1

523:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np3_vs_np1

524:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np4_vs_np1

525:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp
Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5900/scream/components/eamxx && source /projects/sems/modulefiles/utils/sems-modules-init.sh && module purge && module load sems-cmake/3.27.9 sems-git/2.42.0 sems-gcc/11.4.0 sems-openmpi-no-cuda/4.1.6 sems-netcdf-c/4.9.2 sems-netcdf-cxx/4.2 sems-netcdf-fortran/4.6.1 sems-parallel-netcdf/1.12.3 sems-openblas && export GATOR_INITIAL_MB=4000MB && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy

RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5900/scream/components/eamxx && source /projects/sems/modulefiles/utils/sems-modules-init.sh && module purge && module load sems-cmake/3.27.9 sems-git/2.42.0 sems-gcc/11.4.0 sems-openmpi-no-cuda/4.1.6 sems-netcdf-c/4.9.2 sems-netcdf-cxx/4.2 sems-netcdf-fortran/4.6.1 sems-parallel-netcdf/1.12.3 sems-openblas && export GATOR_INITIAL_MB=4000MB && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5900/scream/components/eamxx

mappy failed'

[[ 1 == 0 ]]
[[ mappy == \m\a\p\p\y ]]
[[ 0 == 1 ]]
[[ -z '' ]]
[[ 1 == 1 ]]
[[ 0 == 1 ]]
[[ 1 == 1 ]]

++ ../../cime/scripts/create_test e3sm_scream_v1_at -c -b master --wait

Build timed out (after 120 minutes). Marking the build as failed.

$ ssh-agent -k

unset SSH_AUTH_SOCK;

unset SSH_AGENT_PID;

echo Agent pid 1524890 killed;

[ssh-agent] Stopped.

Build was aborted

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins9377245548128033424.sh

Terminated

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

E3SM-Bot · 2024-10-10T20:40:11Z

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

E3SM-Bot · 2024-10-10T20:43:24Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 6141
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	I/O;AT: RETEST
PULLREQUESTNUM	3005
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`adc4a92`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`c41c698`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5903
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	I/O;AT: RETEST
PULLREQUESTNUM	3005
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`adc4a92`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`c41c698`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: tcclevenger

E3SM-Bot · 2024-10-10T22:50:49Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 6141
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	I/O;AT: RETEST
PULLREQUESTNUM	3005
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`adc4a92`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`c41c698`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5903
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	I/O;AT: RETEST
PULLREQUESTNUM	3005
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`adc4a92`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`c41c698`
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Weaver # 6141 PASSED (click to see last 100 lines of console output)


142/157 Test #142: model_initial .........................................................   Passed    5.73 sec
        Start 143: model_restart
143/157 Test #143: model_restart .........................................................   Passed    6.92 sec
        Start 144: restarted_vs_monolithic_check_np1
144/157 Test #144: restarted_vs_monolithic_check_np1 .....................................   Passed    0.10 sec
        Start 145: homme_shoc_cld_spa_p3_rrtmgp_np1
145/157 Test #145: homme_shoc_cld_spa_p3_rrtmgp_np1 ......................................   Passed   11.58 sec
        Start 146: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp
146/157 Test #146: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp .............................   Passed    0.16 sec
        Start 147: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
147/157 Test #147: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1 ............................   Passed   14.22 sec
        Start 148: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
148/157 Test #148: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1 .................   Passed    1.40 sec
        Start 149: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
149/157 Test #149: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp ...................   Passed    0.65 sec
        Start 150: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np1
150/157 Test #150: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np1 ...............................   Passed   12.66 sec
        Start 151: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp
151/157 Test #151: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp ......................   Passed    0.10 sec
        Start 152: homme_shoc_cld_p3_mam_optics_rrtmgp_np1
152/157 Test #152: homme_shoc_cld_p3_mam_optics_rrtmgp_np1 ...............................   Passed   10.89 sec
        Start 153: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp
153/157 Test #153: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp ......................   Passed    0.16 sec
        Start 154: homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_np1
154/157 Test #154: homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_np1 ............   Passed   11.32 sec
        Start 155: homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_baseline_cmp
155/157 Test #155: homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_baseline_cmp ...   Passed    0.17 sec
        Start 156: homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_np1
156/157 Test #156: homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_np1 .........................   Passed   31.81 sec
        Start 157: homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp
157/157 Test #157: homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp ................   Passed    0.16 sec
100% tests passed, 0 tests failed out of 157
Label Time Summary:

baseline_cmp               = 140.96 secproc (23 tests)

baseline_gen               = 324.93 secproc (25 tests)

bfbhash                    =   0.96 secproc (1 test)

check                      =   0.92 secproc (1 test)

cld                        =  43.23 secproc (7 tests)

cld_fraction               =   4.43 secproc (1 test)

cxx baseline_cmp           =  10.10 secproc (2 tests)

diagnostics                =  45.77 secproc (23 tests)

driver                     =  94.20 secproc (16 tests)

dynamics                   =   5.35 secproc (3 tests)

fail                       =  30.89 secproc (5 tests)

io                         =  51.67 secproc (14 tests)

mam4_aci                   =  29.55 secproc (4 tests)

mam4_constituent_fluxes    =   7.75 secproc (1 test)

mam4_drydep                =   3.45 secproc (1 test)

mam4_optics                =   8.60 secproc (1 test)

mam4_srf_online_emiss      =   7.75 secproc (1 test)

mam4_wetscav               =  21.31 secproc (2 tests)

nudging                    =   8.92 secproc (2 tests)

p3                         = 111.75 secproc (12 tests)

p3_sk                      =  34.30 secproc (2 tests)

physics                    = 190.36 secproc (27 tests)

remap                      =   5.10 secproc (1 test)

rrtmgp                     =  46.95 secproc (11 tests)

shoc                       =  54.56 secproc (13 tests)

spa                        =   7.76 secproc (4 tests)

surface_coupling           =   4.75 sec*proc (1 test)
Total Test time (real) = 800.07 sec
Testing '''d5a54c745b09e934c19025b6ec5b52d195c3b8ed''' for test '''full_sp_debug'''
RUN: taskset -c 52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6141/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6141/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6141/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6141/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.1/gcc/11.3.0/openmpi/4.1.6/5tv5psl -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.2/gcc/11.3.0/openmpi/4.1.6/pyuuqd3 -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.6/2s52shy -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6141/scream/components/eamxx/ctest-build/full_sp_debug
Testing '''d5a54c745b09e934c19025b6ec5b52d195c3b8ed''' for test '''release'''
RUN: taskset -c 104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6141/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6141/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6141/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6141/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.1/gcc/11.3.0/openmpi/4.1.6/5tv5psl -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.2/gcc/11.3.0/openmpi/4.1.6/pyuuqd3 -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.6/2s52shy -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6141/scream/components/eamxx/ctest-build/release
Testing '''d5a54c745b09e934c19025b6ec5b52d195c3b8ed''' for test '''full_debug'''
RUN: taskset -c 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6141/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6141/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6141/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6141/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.1/gcc/11.3.0/openmpi/4.1.6/5tv5psl -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.2/gcc/11.3.0/openmpi/4.1.6/pyuuqd3 -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.6/2s52shy -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DKokkos_ENABLE_DEBUG_BOUNDS_CHECK=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6141/scream/components/eamxx/ctest-build/full_debug

OVERALL STATUS: PASS

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6141/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6141/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6141/scream/components/eamxx

Completed analysis on weaver'

[[ 0 != 0 ]]
[[ 1 == 0 ]]
[[ weaver == \m\a\p\p\y ]]
set +x

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins7527381900174134327.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Finished: SUCCESS

SCREAM_PullRequest_Autotester_Mappy # 5903 FAILED (click to see last 100 lines of console output)


        Start 542: homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_baseline_cmp
549/550 Test #541: homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_np4_vs_np1 .....   Passed    0.35 sec
550/550 Test #542: homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_baseline_cmp ...   Passed    0.40 sec
100% tests passed, 0 tests failed out of 550
Label Time Summary:

PEM                        =  13.93 secproc (90 tests)

baseline_cmp               =  31.60 secproc (23 tests)

baseline_gen               = 12943.65 secproc (25 tests)

bfbhash                    =   1.75 secproc (1 test)

check                      =   1.74 secproc (1 test)

cld                        = 19521.95 secproc (94 tests)

cld_fraction               =  11.65 secproc (1 test)

cosp                       = 111.20 secproc (3 tests)

cxx baseline_cmp           =  62.56 secproc (2 tests)

diagnostics                = 3314.73 secproc (26 tests)

dp                         = 1121.18 secproc (6 tests)

driver                     = 28599.20 secproc (112 tests)

dynamics                   = 11501.16 secproc (66 tests)

fail                       = 1343.79 secproc (35 tests)

io                         = 567.50 secproc (53 tests)

mam4_aci                   = 8471.75 secproc (37 tests)

mam4_constituent_fluxes    = 340.18 secproc (13 tests)

mam4_drydep                = 3184.26 secproc (16 tests)

mam4_optics                = 9560.75 secproc (25 tests)

mam4_srf_online_emiss      = 381.65 secproc (13 tests)

mam4_wetscav               = 3430.62 secproc (26 tests)

nudging                    = 1280.90 secproc (8 tests)

p3                         = 21490.25 secproc (144 tests)

p3_sk                      = 1238.45 secproc (32 tests)

pg2                        = 497.51 secproc (9 tests)

physics                    = 31052.34 secproc (276 tests)

remap                      =  24.18 secproc (4 tests)

rrtmgp                     = 22060.90 secproc (98 tests)

shoc                       = 20629.80 secproc (121 tests)

spa                        = 4001.65 secproc (43 tests)

surface_coupling           =  23.95 secproc (4 tests)

tms                        = 1618.50 secproc (12 tests)
Total Test time (real) = 1742.46 sec
Testing '''6c22f9289d96f1f3da1b76bb9fc987c61d51d225''' for test '''full_sp_debug'''
RUN: taskset -c 32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5903/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5903/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5903/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=mappy -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5903/scream/components/eamxx/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel9-x86_64/sems/tpl/netcdf-fortran/4.6.1/gcc/11.4.0/openmpi/4.1.6/rztypf2 -DNetCDF_C_PATH=/projects/sems/install/rhel9-x86_64/sems/tpl/netcdf-c/4.9.2/gcc/11.4.0/openmpi/4.1.6/po6gjlz -DPnetCDF_C_PATH=/projects/sems/install/rhel9-x86_64/sems/tpl/parallel-netcdf/1.12.3/gcc/11.4.0/openmpi/4.1.6/llgqd5t -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32 -DSCREAM_BASELINES_DIR=/sems-data-store/ACME/baselines/scream/master-baselines/full_sp_debug" '''

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5903/scream/components/eamxx/ctest-build/full_sp_debug
Testing '''6c22f9289d96f1f3da1b76bb9fc987c61d51d225''' for test '''debug_nopack_fpe'''
RUN: taskset -c 64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5903/scream/components/eamxx/ctest-build/debug_nopack_fpe/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5903/scream/components/eamxx/ctest-build/debug_nopack_fpe -DBUILD_NAME_MOD=debug_nopack_fpe -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5903/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=mappy -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5903/scream/components/eamxx/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel9-x86_64/sems/tpl/netcdf-fortran/4.6.1/gcc/11.4.0/openmpi/4.1.6/rztypf2 -DNetCDF_C_PATH=/projects/sems/install/rhel9-x86_64/sems/tpl/netcdf-c/4.9.2/gcc/11.4.0/openmpi/4.1.6/po6gjlz -DPnetCDF_C_PATH=/projects/sems/install/rhel9-x86_64/sems/tpl/parallel-netcdf/1.12.3/gcc/11.4.0/openmpi/4.1.6/llgqd5t -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_PACK_SIZE=1 -DSCREAM_FPE=True -DSCREAM_ENABLE_BASELINE_TESTS=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32 -DSCREAM_ENABLE_BASELINE_TESTS=Off" '''

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5903/scream/components/eamxx/ctest-build/debug_nopack_fpe
Testing '''6c22f9289d96f1f3da1b76bb9fc987c61d51d225''' for test '''release'''
RUN: taskset -c 96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5903/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5903/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5903/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=mappy -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5903/scream/components/eamxx/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel9-x86_64/sems/tpl/netcdf-fortran/4.6.1/gcc/11.4.0/openmpi/4.1.6/rztypf2 -DNetCDF_C_PATH=/projects/sems/install/rhel9-x86_64/sems/tpl/netcdf-c/4.9.2/gcc/11.4.0/openmpi/4.1.6/po6gjlz -DPnetCDF_C_PATH=/projects/sems/install/rhel9-x86_64/sems/tpl/parallel-netcdf/1.12.3/gcc/11.4.0/openmpi/4.1.6/llgqd5t -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32 -DSCREAM_BASELINES_DIR=/sems-data-store/ACME/baselines/scream/master-baselines/release" '''

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5903/scream/components/eamxx/ctest-build/release
Testing '''6c22f9289d96f1f3da1b76bb9fc987c61d51d225''' for test '''full_debug'''
RUN: taskset -c 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5903/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5903/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5903/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=mappy -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5903/scream/components/eamxx/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel9-x86_64/sems/tpl/netcdf-fortran/4.6.1/gcc/11.4.0/openmpi/4.1.6/rztypf2 -DNetCDF_C_PATH=/projects/sems/install/rhel9-x86_64/sems/tpl/netcdf-c/4.9.2/gcc/11.4.0/openmpi/4.1.6/po6gjlz -DPnetCDF_C_PATH=/projects/sems/install/rhel9-x86_64/sems/tpl/parallel-netcdf/1.12.3/gcc/11.4.0/openmpi/4.1.6/llgqd5t -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DKokkos_ENABLE_DEBUG_BOUNDS_CHECK=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32 -DSCREAM_BASELINES_DIR=/sems-data-store/ACME/baselines/scream/master-baselines/full_debug" '''

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5903/scream/components/eamxx/ctest-build/full_debug

OVERALL STATUS: PASS

Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5903/scream/components/eamxx && source /projects/sems/modulefiles/utils/sems-modules-init.sh && module purge && module load sems-cmake/3.27.9 sems-git/2.42.0 sems-gcc/11.4.0 sems-openmpi-no-cuda/4.1.6 sems-netcdf-c/4.9.2 sems-netcdf-cxx/4.2 sems-netcdf-fortran/4.6.1 sems-parallel-netcdf/1.12.3 sems-openblas && export GATOR_INITIAL_MB=4000MB && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy

RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5903/scream/components/eamxx && source /projects/sems/modulefiles/utils/sems-modules-init.sh && module purge && module load sems-cmake/3.27.9 sems-git/2.42.0 sems-gcc/11.4.0 sems-openmpi-no-cuda/4.1.6 sems-netcdf-c/4.9.2 sems-netcdf-cxx/4.2 sems-netcdf-fortran/4.6.1 sems-parallel-netcdf/1.12.3 sems-openblas && export GATOR_INITIAL_MB=4000MB && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5903/scream/components/eamxx

Completed analysis on mappy'

[[ 0 != 0 ]]
[[ 1 == 0 ]]
[[ mappy == \m\a\p\p\y ]]
[[ 0 == 1 ]]
[[ -z '' ]]
[[ 1 == 1 ]]
[[ 0 == 1 ]]
[[ 1 == 1 ]]

++ ../../cime/scripts/create_test e3sm_scream_v1_at -c -b master --wait

Build timed out (after 120 minutes). Marking the build as failed.

$ ssh-agent -k

unset SSH_AUTH_SOCK;

unset SSH_AGENT_PID;

echo Agent pid 485011 killed;

[ssh-agent] Stopped.

Build was aborted

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins1503897465682789180.sh

Terminated

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

tcclevenger · 2024-10-15T22:18:56Z

@bartgol I finally found the issue, it was with changing the constructor to be non-default. Looking at the list of member variable, nothing is popping out as being the problem being un-initialized.

I chose to keep the default constructor and add an OutputManager::initialize() function. If it is preferred I can hunt down the problem variable and put initialization back in the class constructor.

E3SM-Bot · 2024-10-17T15:42:16Z

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

E3SM-Bot · 2024-10-17T15:42:20Z

Status Flag 'Pull Request AutoTester' - GitHub reports Mergeable status = False

(passing FM and GM)

E3SM-Bot · 2024-10-17T20:13:15Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 6175
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	I/O
PULLREQUESTNUM	3005
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`8785b05`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`890d5d7`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5927
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	I/O
PULLREQUESTNUM	3005
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`8785b05`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`890d5d7`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: tcclevenger

E3SM-Bot · 2024-10-17T21:26:18Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 6175
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	I/O
PULLREQUESTNUM	3005
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`8785b05`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`890d5d7`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5927
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	I/O
PULLREQUESTNUM	3005
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`8785b05`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`890d5d7`
TEST_REPO_ALIAS	SCREAM

E3SM-Bot · 2024-10-17T21:26:35Z

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
THE LAST COMMIT TO THIS PULL REQUEST HAS NOT BEEN REVIEWED YET!

E3SM-Bot · 2024-10-17T21:26:41Z

All Jobs Finished; status = PASSED, target_sha=890d5d7778f8882628c67cf05f406bd859c0c0b4, However Inspection must be performed before merge can occur...

E3SM-Bot · 2024-10-18T14:42:45Z

The base branch has been updated since the last successful testing.

last PASS base branch sha: 890d5d7
current base branch sha : 624e53c
The AutoTester will discard the last PASS, and re-test the PR from scratch

E3SM-Bot · 2024-10-18T14:45:48Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 6177
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	I/O;AT: AUTOMERGE
PULLREQUESTNUM	3005
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`8785b05`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`890d5d7`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5929
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	I/O;AT: AUTOMERGE
PULLREQUESTNUM	3005
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`8785b05`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`890d5d7`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: tcclevenger

E3SM-Bot · 2024-10-18T15:57:58Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 6177
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	I/O;AT: AUTOMERGE
PULLREQUESTNUM	3005
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`8785b05`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`890d5d7`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5929
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	I/O;AT: AUTOMERGE
PULLREQUESTNUM	3005
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`8785b05`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`890d5d7`
TEST_REPO_ALIAS	SCREAM

E3SM-Bot · 2024-10-18T15:58:15Z

All Jobs Finished; status = PASSED, target_sha=624e53c81ea2d950f8520e33a209e14cbba0fec1, However Inspection must be performed before merge can occur...

tcclevenger added the I/O label Sep 18, 2024

tcclevenger requested review from bartgol, AaronDonahue and mahf708 September 18, 2024 22:28

tcclevenger self-assigned this Sep 18, 2024

bartgol reviewed Sep 18, 2024

View reviewed changes

tcclevenger added the AT: WIP label Sep 19, 2024

tcclevenger changed the title ~~I/O: Pass output parameter list to processes~~ I/O: Split OutputManager creation from setup Oct 9, 2024

tcclevenger force-pushed the tcclevenger/split_output_setup branch from 630fdc6 to 7cc2511 Compare October 9, 2024 18:55

tcclevenger requested a review from bartgol October 9, 2024 18:57

tcclevenger added AT: RETEST and removed AT: WIP labels Oct 9, 2024

E3SM-Bot removed the AT: RETEST label Oct 9, 2024

tcclevenger added AT: WIP and removed AT: WIP labels Oct 9, 2024

tcclevenger added the AT: WIP label Oct 10, 2024

E3SM-Bot removed the AT: RETEST label Oct 10, 2024

tcclevenger force-pushed the tcclevenger/split_output_setup branch 2 times, most recently from 56b7a77 to 6245275 Compare October 15, 2024 21:54

tcclevenger removed the AT: WIP label Oct 15, 2024

bartgol added the AT: RETEST label Oct 17, 2024

bartgol mentioned this pull request Oct 17, 2024

Side bar not docked when browser zoom is too large refined-github/refined-github#7925

Closed

1 task

E3SM-Bot removed the AT: RETEST label Oct 17, 2024

tcclevenger dismissed bartgol’s stale review via 77602fe October 17, 2024 17:17

tcclevenger added 10 commits October 17, 2024 13:53

Split construction of OM (passing comm,params,ts,is_restart) with setup

7222540

(passing FM and GM)

Call create_output_managers() for tests that do not use AD::initialize()

0830d9d

Add initalize function, keep constructor defaulted

543a04e

Move set_provenence_data call to in front of create_output_managers

92082e9

fix incorrect inputs

36e4718

Fix run_t0 and case_t0 checks

7385397

Only call set_logger() once

06cace5

Separate restart output manager from output manager list

36d63a3

Improve OutputManager function descriptions

14c1654

Rename set_params -> setup_internals

8785b05

tcclevenger force-pushed the tcclevenger/split_output_setup branch from 77602fe to 8785b05 Compare October 17, 2024 19:55

tcclevenger added the AT: AUTOMERGE label Oct 17, 2024

bartgol approved these changes Oct 18, 2024

View reviewed changes

E3SM-Bot merged commit 0fb8150 into master Oct 18, 2024
6 checks passed

E3SM-Bot removed the AT: AUTOMERGE label Oct 18, 2024

E3SM-Bot deleted the tcclevenger/split_output_setup branch October 18, 2024 17:43

I/O: Split OutputManager creation from setup #3005

I/O: Split OutputManager creation from setup #3005

Conversation

tcclevenger commented Sep 18, 2024 • edited Loading

bartgol left a comment

Choose a reason for hiding this comment

bartgol Sep 18, 2024

Choose a reason for hiding this comment

E3SM-Bot commented Sep 19, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Using Repos:

E3SM-Bot commented Sep 19, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

E3SM-Bot commented Oct 9, 2024

E3SM-Bot commented Oct 9, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Using Repos:

E3SM-Bot commented Oct 9, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs

E3SM-Bot commented Oct 10, 2024

E3SM-Bot commented Oct 10, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Using Repos:

E3SM-Bot commented Oct 10, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Total Test time (real) = 800.07 sec

Testing '''d5a54c745b09e934c19025b6ec5b52d195c3b8ed''' for test '''full_sp_debug'''

Testing '''d5a54c745b09e934c19025b6ec5b52d195c3b8ed''' for test '''release'''

Testing '''d5a54c745b09e934c19025b6ec5b52d195c3b8ed''' for test '''full_debug'''

Total Test time (real) = 1742.46 sec

Testing '''6c22f9289d96f1f3da1b76bb9fc987c61d51d225''' for test '''full_sp_debug'''

Testing '''6c22f9289d96f1f3da1b76bb9fc987c61d51d225''' for test '''debug_nopack_fpe'''

Testing '''6c22f9289d96f1f3da1b76bb9fc987c61d51d225''' for test '''release'''

Testing '''6c22f9289d96f1f3da1b76bb9fc987c61d51d225''' for test '''full_debug'''

We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs

tcclevenger commented Oct 15, 2024

E3SM-Bot commented Oct 17, 2024

E3SM-Bot commented Oct 17, 2024

E3SM-Bot commented Oct 17, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Using Repos:

E3SM-Bot commented Oct 17, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

E3SM-Bot commented Oct 17, 2024

E3SM-Bot commented Oct 17, 2024

tcclevenger commented Sep 18, 2024 •

edited

Loading