Meeting Notes April 2020

Wednesday, April 8, 2020

Attendees - Rusty Davis (rstyd), Pat Grubel (pagrubel), Al McPherson (mcpherson), Tim Randles (trandles-lanl), Jake Tronge (jtronge)

Agenda

1. discuss CWL and container support
2. TaskManager design for modular support of container runtimes and resource managers
3. discuss proposed FY21 ECP P6 Activites
   • BEE- FY21 P6-1 Develop the ability to archive, clone, and re-run workflows (start 10/01/20, due 3/31/21)
   • BEE- FY21 P6-2 Run BEE jobs on private cloud infrastructure (due 9/31/21)

PR Review

1. NONE

Issue Review

1. NONE

Discussion

1. APPROVED April 6, 2020 meeting notes
2. TaskManager discussion mostly shelved for now, revisit next week
3. CWL support for containers
   • defined at https://www.commonwl.org/v1.1/CommandLineTool.html#DockerRequirement
   • how to handle bind mounts, inputs/outputs, etc.?
     • maybe a question for the CWL mailing list
     • Rusty looking for existing containerized CWL workflows for examples
   • "standard" container runtime options in the bee.conf file?
4. FY21 ECP Activities are documented at
   • Tim starting on design document for the activities

Around the room

• Jake
  • neo4j issues (Task already exists)
  • Rusty knows how to fix itj
  • close to being able to run test workflows
• Rusty
  • starting test work
  • looking at PyTest for integration testing
  • maybe pexpect for client testing
  • Flask has some testing framework (Jake)
  • BEE should start a document of what CWL is supported by project
• Pat
  • question for Rusty about passing Task object to worker from TaskManager
  • will need to think about how to pass things around when there's more data (requirements and hints)
• Al
  • refactoring database and building new API to it
    • no way to version python APIs
    • API changes only affect WorkflowManager
  • next use case CWL example
    • maybe BLAST workflow again
    • keep scope of parsing to HPC use cases, not "generic everything CWL"
    • Do srudin VASP workflow (parameter study) (issue #66)

Action Items

1. Tim - get VASP containers that work with Charliecloud (Power9, x86_64)

---

Monday, April 6, 2020

Attendees - Rusty Davis (rstyd), Pat Grubel (pagrubel), Qiang Guan (guanxyz), Tim Randles (trandles-lanl), Jake Tronge (jtronge)

PR Review

1. #138 APPROVED (trandles-lanl) - Use bee.conf to configure listen ports for BEEWorkflowManager and BEETaskManager
   • Pat approves of merging this PR, but into master instead of develop. The rationale is the functionality is simple and enables everyone to do development work at the same time on the same system.

Issue Review

1. #137 (pagrubel) - Slurm worker to properly check DockerRequirment
   • slurm_worker.py should use the DockerRequirement: dockerImageId specified in the CWL file

Discussion

1. extending CWL for other container runtimes (rstyd)
   • discuss on Wednesday
   • guanxyz had some ideas
2. next ECP milestones up on wiki

Around the room

• Jake
  • got a test environment set up at KSU
  • initial problems with PySlurm due to having a too-new Slurm installed
• Rusty
  • working on unittest and CI tests for client/WorkflowManager
  • not a lot of time for BEE this week (very understandable, everyone prioritized BEE the past 2 week (trandles-lanl))
• Pat
  • unittest for TaskManager
  • issue #137 above
  • not much time for BEE this week
• Tim
  • issue #139 planning to discuss on Wednesday
  • ECP milestone housekeeping