Skip to content

DUNE DAQ Suite v2 progress tracking

Jump to bottom
Alessandro Thea edited this page Nov 23, 2020 · 5 revisions

Release Objectives

The v2.0 release primary focus is the daq applications command interface and handling of the associated data. These are deliverables of the Command and Configuration subsystem of CCM.

  1. Introduce a new interface to deliver commands to application/modules with a parameters defined via a schema.
  2. Provide tools to:
    • Define a command parameter set through a schema
    • Generate the corresponding C++ containers and the code to serialise/deserialize them form the raw format (json)
    • Create command configuration files
    • Validate application/module configuration against schema
  3. Consolidate daq package structure and cmake files.

Table of Content

Packages

Package Current Branch/Tag Expected update Reference Documentation Responsible
daq-buildtools 1.3.0 package versions updates only (daq-cmake) build instructions John, Alessandro
daq-cmake 1.0.0 moo cmake intergration cmake macros John, Alessandro
moo 0.4.1 oschema functions Brett
ers v1.0.0 none Alessandro
cmdlib v1.0.1 none Roland
restcmd v1.0.1 none Roland
filecmd master ? Brett
appfwk develop schema/ folder rationalisation cmd schema funcs Alessandro, Marco, Eric
listrev develop schema/ folder rationalisation Kurt

Postponed

Package Current Branch/Tag Expected update Reference Documentation Responsible
opmonlin master Lola

Command data model

Release v2.0 introduces a command data model for DAQ application commands exploiting the concept of schema. The moo python package provides library functions that help make the creation of schema files a little easier, and it provides the tools to generate code and configuration based on the schema.
The associated deliverables in v2.0 belong to two different groups:

  1. Schema definition and (C++) code generation
  2. Support for creating daq module and application configuration / test files.

The distinction between the two is discussed in details in Kurt's presentation at the Nov. 16 coordination meeting. For each group, the deliverables consist of the tools (moo + specific daq extensions) to apply the model to DAQ Modules and the supporting documentation.

Schema and model creation

In terms of tools, moo has been used or a while by now, to demonstrate schema definition and code generation. On the other hand the documentation available doesn't seem to clearly explain at the appropriate level of detail
what goes into creting a schema such that a developer has a control of every step.

Instruction for creating schema/model files is avalable on the moo website:

https://brettviren.github.io/moo/dunedaq-appfwk-schema.html

Similar information seem to available in the README.org in appfwk/schema.

Key questions:

  • "How to create a new schema file":
    Both documents contain a fair mount of information, but with a certain lack of focus on some details:

    • The schema definition structure section is confusing for people unfamiliar with jsonnet.
    • It is not clear if all the types made available by moo are suitable for DAQ modules.
      • numpy types are used in the example. Are they all usable or do some of them don't play well with code generation?
    • A complete schema example is missing (separated preamble and body)
    • Why the need for a sorted list of types? Could it handled behind the scenes?

  • "Model files, what are they?" Note: this may be outdated. Model files seem to have disappeared in more recent examples than appfwk's

    • Model files are present in appfwk/schema, but are not mentioned in dunedaq-appfwk-schema (is that right?)
    • appfwk/schema/appfwk-*-model.jsonnet seem to have the only role to "join" appfwk-model.jsonnet with the specific model file. Is this needed?
    • General lack of explanation of the role of model files.

  • "How to generate command data C++ code?" Note: while this stage is supposed to be hidden from user by cmake modules, information should be made available on the nderlying use of moo render and what ingredients (model, schema and template files) go into it

    • The example available in the above document seems to have fallen behind appfwk's generate.sh example.
      An up-to-date example is needed.
    • Also, the use of moo render in appfwk and other examples (listrev, ddpdemo) are not consistent. Which one is an up-to-date reference?

Command data object creation

<To do >

Command facility infrastructure

The goal for the command facility infrastructure in the v2.0 context is to provide simple and yest solid mechanism to send commands to daq applications and modules, to support their development. The requirements of the command user interface are

  1. Define a command file format that describes a list of comamnd and the corresponding parameters to be sent to a daq application
  2. Support for interactive mode and automatic command injection
  3. Desirable: support for (python) scripting
  4. Clear user instructions and recommendations

cmdlib

  • Tagged (v1.0.1) and ready for release
  • docs/UserGuide.md provides ke information on the command interface: file format, C++ interface and command facility.

restcmd

  • Tagged (v1.0.1) and ready for release (but...)
  • Building and running information available in README.md
    • But build instructions in README.md still refer to daq-buildtools v1.2.1.
    • Also, pistache has been moved to the main product area and included in the set of packages included in daq-buildtools v1.3.0 areas.

filecmd

  • Documentation consolidation is needed.

    • Missing "getting started" informations.
      Some documentation is available as part of ddrepl link but should be made available in the repo README or in a separatedoc/ document.
    • Missing specifications of command files. Clarify whether it's the same format used by stdinCommandFacility
      • Differences between json and jsonstream?

  • Clarify the role of ddrepl: external tool or should be moved into the repo?

    • if external, clarfify the scope.

  • Tag still pending. Current package version 0.0.0

    project(restcmd VERSION 0.0.0)

Monitoring Interface

  • Code available at https://github.com/Lola-Stankovic/opmonlib
  • Includes file and influxdb backend. Influx's to be moved to different repo (plugin)
  • Introduce a standard DAQModule interface to automatically provide for the right context.
  • Inclusion postponed to a patch release.

Buildtools

  • Python environment added to daq-builtools development area
  • Cmake modules split from daq-builtools
  • moo integration with daq-cmake functions
Clone this wiki locally