Trouble in running recipe_ocean_example.yml

[helenelang]

Hi!

I’m trying to run the script recipe_ocean_example.yml on Levante/DKRZ. I’ve tried to run the script in both work and scratch, but the script seems to hang and will not complete. I’ve run the
same script before on Mistral (under work) without any problems.

The last days I've tried to use the Levante job scheduler, to see if this can solve the problem. Still, unfortunately, the script will not complete.

I would appreciate very much your help on how to solve this problem.

Below is both the shell script, the ESMValtool recipe, and the main_log_debug file is attached.

Thanks,
Helene

main_log_debug.txt

#!/bin/bash
#SBATCH -J myjob             # Specify job name
#SBATCH -p shared           # Use partition shared
#SBATCH --mem=50G      # Specify memory amount needed
#SBATCH -t 15:00:00        # Set a limit on the total run time
#SBATCH -A bk1088        # Charge resources on this project account
#SBATCH -o myjob.o%j     # File name for standard and error output

set -e

module load esmvaltool/2.7.0

esmvaltool run /scratch/b/b381145/recipe_ocean_example.yml

# ESMValTool
# recipe_ocean_example.yml
---
documentation:
  title: Ocean physics analysis suite
  
  description: |
    Recipe to demonstrate several simple plots based on the monthly ocean
    temperature. Please use this file as a template for adding additional
    fields into the ocean. This work based on the BGC-val toolkit GMD-2018-103.

  authors:
    - demora_lee

  maintainer:
    - demora_lee

  references:
    - demora2018gmd

  projects:
    - ukesm


datasets:
  # working datasets
  # - {dataset: CanESM2, project: CMIP5, mip: Omon, exp: historical, ensemble: r1i1p1, start_year: 2001, end_year: 2004}
  # - {dataset: GISS-E2-H, project: CMIP5, mip: Omon, exp: historical, ensemble: r1i1p1, start_year: 2001, end_year: 2004}
  # - {dataset: HadGEM2-AO, project: CMIP5, mip: Omon, exp: historical, ensemble: r1i1p1, start_year: 2001, end_year: 2004}
  # - {dataset: HadCM3, project: CMIP5, mip: Omon, exp: historical, ensemble: r1i1p1, start_year: 2001, end_year: 2004}
  # - {dataset: CanCM4, project: CMIP5, mip: Omon, exp: historical, ensemble: r1i1p1, start_year: 2001, end_year: 2004}
  # - {dataset: CSIRO-Mk3-6-0, project: CMIP5, mip: Omon, exp: historical, ensemble: r1i1p1, start_year: 2001, end_year: 2004}
  
  # - {dataset: CanESM5-CanOE, project: CMIP6, mip: Omon, exp: [historical, ssp126], ensemble: r1i1p1f1, grid: gn, start_year: 1993, end_year: 2025}



# FUTURE (only ssp)
 # - {dataset: EC-Earth3, project: CMIP6, mip: Omon, exp: ssp585, ensemble: r1i1p1f1, grid: gn, start_year: 2026, end_year: 2058}
 # - {dataset: MRI-ESM2-0, project: CMIP6, mip: Omon, exp: ssp585, ensemble: r1i1p1f1, grid: gr, start_year: 2026, end_year: 2058}
 # - {dataset: CIESM, project: CMIP6, mip: Omon, exp: ssp585, ensemble: r1i1p1f1, grid: gn, start_year: 2026, end_year: 2058}
 # - {dataset: IPSL-CM6A-LR, project: CMIP6, mip: Omon, exp: ssp585, ensemble: r1i1p1f1, grid: gn, start_year: 2026, end_year: 2058}
 # - {dataset: INM-CM5-0, project: CMIP6, mip: Omon, exp: ssp585, ensemble: r1i1p1f1, grid: gr1, start_year: 2026, end_year: 2058}
 # - {dataset: MPI-ESM1-2-HR, project: CMIP6, mip: Omon, exp: ssp585, ensemble: r1i1p1f1, grid: gn, start_year: 2026, end_year: 2058}
 # - {dataset: CNRM-ESM2-1, project: CMIP6, mip: Omon, exp: ssp585, ensemble: r1i1p1f2, grid: gn, start_year: 2026, end_year: 2058}
 # - {dataset: CanESM5, project: CMIP6, mip: Omon, exp: ssp585, ensemble: r1i1p1f1, grid: gn, start_year: 2026, end_year: 2058}
 # - {dataset: CAMS-CSM1-0, project: CMIP6, mip: Omon, exp: ssp585, ensemble: r1i1p1f1, grid: gn, start_year: 2026, end_year: 2058}
 # NOW WORKING! Before: Something wrong with CESM: “Generic level coordinate olevel with out_name lev has wrong standard_name or is not set”
 # - {dataset: CESM2-WACCM, project: CMIP6, mip: Omon, exp: ssp126, ensemble: r1i1p1f1, grid: gr, start_year: 2026, end_year: 2058}
  - {dataset: NorESM2-MM, project: CMIP6, mip: Omon, exp: ssp585, ensemble: r1i1p1f1, grid: gr, start_year: 2026, end_year: 2058}
 # - {dataset: GFDL-ESM4, project: CMIP6, mip: Omon, exp: ssp585, ensemble: r1i1p1f1, grid: gr, start_year: 2026, end_year: 2058}
 # - {dataset: MPI-ESM1-2-LR, project: CMIP6, mip: Omon, exp: ssp585, ensemble: r1i1p1f1, grid: gn, start_year: 2026, end_year: 2058}
# COMBINED (hist+ssp):
 # - {dataset: MPI-ESM1-2-LR, project: CMIP6, mip: Omon, exp: [historical, ssp126], ensemble: r1i1p1f1, grid: gn, start_year: 1993, end_year: 2025}



  # - {dataset: HadGEM2-CC, project: CMIP5, mip: Omon, exp: historical, ensemble: r1i1p1, start_year: 2001, end_year: 2004}
  # - {dataset: HadGEM2-ES, project: CMIP5, mip: Omon, exp: historical, ensemble: r1i1p1, start_year: 2001, end_year: 2004}

  #####
  # Data with Errors right now -June 2018
  ###
  # Problem with times
#  - {dataset: MIROC-ESM, project: CMIP5, mip: Omon, exp: historical, ensemble: r1i1p1, start_year: 2001, end_year: 2004}
#  - {dataset: MIROC-ESM-CHEM, project: CMIP5, mip: Omon, exp: historical, ensemble: r1i1p1, start_year: 2001, end_year: 2004}

  # Unstructured grids
 # - {dataset: MPI-ESM-LR, project: CMIP5, mip: Omon, exp: historical, ensemble: r1i1p1, start_year: 2001, end_year: 2003}
 # - {dataset: MPI-ESM-MR, project: CMIP5, mip: Omon, exp: historical, ensemble: r1i1p1, start_year: 2001, end_year: 2004}
#  - {dataset: ACCESS1-0, project: CMIP5, mip: Omon, exp: historical, ensemble: r1i1p1, start_year: 2001, end_year: 2004}
#  - {dataset: ACCESS1-3, project: CMIP5, mip: Omon, exp: historical, ensemble: r1i1p1, start_year: 2001, end_year: 2004}

# TODO: All multi-model statistics are turned off until the preprocessor order is fixed.

preprocessors:
  # --------------------------------------------------
  # Map preprocessors - 2D fields
  # --------------------------------------------------
  prep_map_3: # For specific levels of 3D fields with regrid
    custom_order: true
    extract_levels:
      levels: [0., 20., 40., 60., 80., 100., 200., 300., 400., 500., 600., 700., 800., 900., 1000., 1100., 1200., 1300., 1400., 1500., 2000., 2500., 3000.,]
      scheme: linear_extrapolate
    annual_statistics:
      operator: mean
    regrid:
      target_grid: 1x1
      scheme: linear
    climate_statistics:
      operator: mean
  #  multi_model_statistics:
  #    span: overlap
  #    statistics: [mean ]


diagnostics:
  # --------------------------------------------------
  # Map diagnostics
  # --------------------------------------------------
  diag_map_3:
    description: Global Ocean Depth Layers regridded map
    variables:
      thetao: # Temperature/salt ocean 3D
        preprocessor: prep_map_3
    scripts:
      Global_Ocean_DepthLayers_regrid_map: &Global_Ocean_DepthLayers_regrid_map
        script: ocean/diagnostic_maps.py

[helenelang]

The recipe looks very strange here, so I attach a pdf of the yml file.
Sorry for this confusion,
Helene

recipe_ocean_example.pdf

[remi-kazeroni]

Hi @helenelang, thanks for reporting this issue. It seems that DKRZ as recently changed their policy so that memory intensive pre/post-processing jobs could no longer be run on login nodes. That might explain why some of your jobs no longer work the way they used to.
Looking at the main_log_debug.txt that you provided, I think your recipe is too memory intensive to be run without using the SLURM scheduler. I have managed to run your recipe successfully on a compute node with the esmvaltool/2.7.0 in order to assess the resources required:

Time for running the recipe was: 0:07:40.490276
Maximum memory used (estimate): 76.1 GB

In order to get a successful run with your batch script, I would suggest to increase the memory requirements: #SBATCH --mem=100G and reduce the max_parallel_tasks: 1 in your config-user.yml. It doesn't look that the recipe requires multiple tasks to be run in parallel. Also, you could reduce the time to reduce the waiting time in the queue: #SBATCH -t 1:00:00

P.S.: to write large code blocks on GitHub, you can make use of ``` delimiters

[helenelang]

Many thanks for your help! This solved my problem – the recipe runs now successfully.

Thanks again,
Helene

[remi-kazeroni]

Great! Happy to hear that solved the issue, thanks for reporting. 👍