How to write a fragment generator

n.b. Before reading this section, please first read the artdaq-demo Overview. Also potentially useful will be the UML class diagrams shown at the bottom of this page

Intro

As mentioned in this wiki’s Overview, artdaq-demo provides an example of a fragment generator, ToySimulator, which developers of artdaq-based DAQ systems can study and use as a starting-off point for writing their own fragment generators. In this section of the wiki, we’ll start by discussing what a fragment generator is, before studying the ToySimulator code and later looking more in-depth into the components it relies on. While reading this document, it will be helpful to look at the artdaq-core-demo and artdaq-demo source code, found in the subdirectories of the same name relative to BASE_DIR/srcs, where “BASE_DIR” is the base directory in which you ran artdaq-demo’s quick-mrb-start.sh script using the [[Installing and building the demo]] instructions. To begin with, of particular interest will be the ToySimulator code, artdaq-demo/artdaq-demo/Generators/ToySimulator.hh and artdaq-demo/artdaq-demo/Generators/ToySimulator_generator.cc as well as the interface to the (fake) hardware, artdaq-demo/artdaq-demo/Generators/ToyHardwareInterface/ToyHardwareInterface.hh, provided as an example of a vendor-supplied API to an experiment’s hardware.

An overview on fragment generators

In the case of the BoardReader, individual experiments develop one or more “FragmentGenerator” plugins that communicate with electronics hardware. The BoardReader code takes care of communicating with Run Control, sending the data to the EventBuilders, etc. And, it calls the appropriate methods in its FragmentGenerator based on the commands that it receives from Run Control. The number of BoardReaders in a DAQ system is easily re-configurable, and typically there is one BoardReader per electronics module.

The task of creating a FragmentGenerator boils down to creating a C class that implements a well-defined set of methods. The characteristics of such a class include the following:

It should inherit from CommandableFragmentGenerator (which defines the expected interface and provides some utility methods)
Its constructor should accept a fhicl::ParameterSet that contains all of the configuration data that is needed, both for configuring itself and all upstream hardware, firmware, and software. And, the bulk of such configuration should be done in the constructor. FragmentGenerators are constructed when BoardReaders received the “init” (i.e. “initialization” or “configuration” message from Run Control).
It should implement “start”, “stop”, and “stopNoMutex” methods. These methods are called when a data taking run begins (start) and when a run is ended (stopNoMutex and stop). The stopNoMutex method can be used to send end-of-run commands to the hardware that are asynchronous with the readout of the data. (The stop command is never sent while the “getNext_” method [described below] is active.)
It should implement a “getNext*” method that reads out the data and formats the data from each “event” (experiment-specific definition) into an artdaq::Fragment. A couple of notes on the getNext* method:
- its argument is a vector of artdaq::Fragments that is used to return zero, one, or more fragments of data
- its return code is a boolean that indicates whether data taking has finished or not. In this case, “data taking” means a data taking run, not just an individual event. Generally, a getNext_ method returns “false” when A) an end-run has been requested and B) existing data in the pipeline has been processed.
- it should not block forever (or even for very long). It is quite acceptable to return a zero-length array of artdaq::Fragments and a return code of “true”. This means that there is no data available at the moment, and the BoardReader should call “getNext*” again. A non-blocking getNext* method allows runs to be ended gracefully even when no data is flowing.
- as part of creating the artdaq::Fragments, getNext_ methods need to create and fill a “sequence ID” in the artdaq::Fragment header. This integer uniquely identifies each fragment from a given BoardReader, and it is used to route the fragment to the correct EventBuilder. The getNext_ method also fills a “fragment ID” field in the artdaq::Fragment header, and this integer identifies the source of the fragment (which BoardReader and which physical part of the detector). In a smoothly running system, the BoardReaders generate one, and only one, fragment with a given combination of sequenceID and fragmentID per run. EventBuilders group fragments with the same sequenceID into full events. The determination of sequenceIDs is experiment-dependent, but of course, the different types of FragmentGenerator used by an experiment must use consistent methods for assigning sequenceIDs so that full events can successfully be built.
There are other operations that FragmentGenerators can implement, but those are optional, and are described elsewhere.

What code gets executed during state transitions

If you’ve already run the demo as described in [[Running a sample artdaq-demo system]], you’ll have a feel for the state model of artdaq-based systems; in short, the (primary) transitions they support are initialize, start, stop and shutdown. In order to write a fragment generator for an artdaq based system, you’ll need to understand what parts of the fragment generator are executed during what transition:

initialize: The fragment generator object is created - i.e., its constructor is called. The constructor should take as an argument a FHiCL parameter set which will be used both to set member values in the fragment generator itself as well as its CommandableFragmentGenerator base class.

start : The fragment generator’s start() function is called. An example of how this function might be used would be to use the hardware’s vendor-supplied API to signal to the hardware that it can start sending data. After this point, the DAQ is in the “running” state, at which point the getNext_() function is called repeatedly.

stop : The fragment generator’s stop() function is called. Here, e.g., one could signal to the hardware to stop sending data. After this, calls to getNext_() cease.

shutdown : The fragment generator object is destroyed - i.e., its destructor is called.

A quick overview of the ToyHardwareInterface API

ToyHardwareInterface is a class meant to mimic a vendor-supplied API for hardware, and is used by ToySimulator accordingly. This API was written in C+03 to reflect that many (most?) vendor-supplied APIs in use haven’t yet started using C+11. A description of the functions it provides (and which are used by ToySimulator) follows; note than “hardware” below doesn’t refer to literal hardware, but rather the imaginary hardware we pretend the API interfaces to:

Signal to the upstream hardware to start / stop sending data:

  void StartDatataking();
  void StopDatataking();

Allocate / deallocate the memory buffer which will be used to store the hardware’s data- to be used in place of new/delete:

 void AllocateReadoutBuffer(char** buffer);
 void FreeReadoutBuffer(char* buffer);

Once the memory buffer is allocated, pass it to this function to be filled with the hardware’s data; the function also returns the number of bytes read:

void FillBuffer(char* buffer, size_t* bytes_read);

In order: provide the board’s serial number, the number of bits representing an ADC value, and the board type:

int SerialNumber() const;
int NumADCBits() const;
int BoardType() const;

A breakdown of ToySimulator

With a basic understanding of what parts of ToySimulator are executed at different stages of datataking and the hardware API it will be working with, we can examine its code.

Constructor:

In the body of the constructor, three main things happen:

a buffer (pointed to by the readout_buffer_ member) has memory allocated in which upstream hardware data will be stored
vendor-supplied info gets stored in the member metadata_ struct
Based on the vendor-supplied board type value, we set the fragment_type_

  hardware_interface_->AllocateReadoutBuffer(&readout_buffer_);

  metadata_.board_serial_number = hardware_interface_->SerialNumber();
  metadata_.num_adc_bits = hardware_interface_->NumADCBits();

  switch (hardware_interface_->BoardType()) {
  case 1002:
    fragment_type_ = toFragmentType("TOY1");
    break;
  case 1003:
    fragment_type_ = toFragmentType("TOY2");
    break;
  default:
    throw cet::exception("ToySimulator") << "Unable to determine board type supplied by hardware";
  }

start():

Hardware is told to start sending data

hardware_interface_->StartDatataking();

getNext()_:

(Some in-code comments stripped out for clarity)

If a “stop” transition has been issued to the DAQ, simply return false; we’re done obtaining data
Otherwise, fill readout_buffer_ with data from the hardware; the hardware API provides us with the # of bytes read in
Use the artdaq::Fragment’s FragmentBytes factory function to create a new artdaq::Fragment. In order, the arguments define the size of the fragment payload, its sequence ID, fragment ID, fragment type, metadata and a timestamp for the fragment. Details will be given in the next document.
Use the classic “memcpy” function to copy the contents of the buffer into the beginning of the artdaq fragment’s payload
Put the artdaq fragment into the frags vector (passed-by-reference to getNext_()) and do this without unnecessarily copying memory
If we have a metric manager object, broadcast how many fragments have been sent
Increment the event counter, used to set the sequence ID in the FragmentBytes function

 if (should_stop()) {
    return false;
  }

  std::size_t bytes_read = 0;
  hardware_interface_->FillBuffer(readout_buffer_ , &bytes_read);                                                                                                     

  std::unique_ptr<artdaq::Fragment> fragptr(
                                            artdaq::Fragment::FragmentBytes(bytes_read,
                                                                            ev_counter(), fragment_id(),
                                                                            fragment_type_,
                                                                                metadata_, timestamp_));

  memcpy(fragptr->dataBeginBytes(), readout_buffer_, bytes_read );

  frags.emplace_back( std::move(fragptr ));

  if(metricMan_ != nullptr) {
    metricMan_->sendMetric("Fragments Sent",ev_counter(), "Events", 3);
  }

  ev_counter_inc();
  timestamp_ += timestampScale_;

  return true;

stop():

Hardware is told to stop sending data

hardware_interface_->StopDatataking();

Destructor:

Deallocate the buffer used to hold upstream data

hardware_interface_->FreeReadoutBuffer(readout_buffer_);

More features

Using separate threads for sending fragments downstream to eventbuilders vs. extraction of data from upstream hardware

One feature we found experiments repeatedly implementing which we consequently added support for was the option to use a separate thread for calls to getNext() - the non-virtual member function of CommandableFragmentGenerator which calls getNext*() and sends the fragments received from getNext*() downstream - vs. the code used to extract data from hardware (i.e., the code in getNext*() ). This feature is set if the “separate_data_thread” parameter, which defaults to “false”, is set to “true”. The advantage of this approach is that backpressure originating downstream won’t automatically slow down the reception of data from hardware- rather, the fragment generator will continue receiving data from upstream via the getNext*() function while buffering the fragments getNext_() returns until it can send the fragments downstream. The buffering will work until either the number of buffered fragments exceeds the value set for the parameter “data_buffer_depth_fragments” or the number of mebibytes (1024**2 bytes) those fragments consist of exceeds the value of the parameter “data_buffer_depth_mb”. Both of these parameters have a default value of 1000 in artdaq v1_13_02. So, to use two separate such threads, and ensure that no more than 100 mebibytes of fragments could be buffered, set the following parameters:

separate_data_thread: true
data_buffer_depth_mb: 100

An option to monitor hardware

The following function has been added to CommandableFragmentGenerator.hh :

// Check any relavent hardware status registers. Return false if                       
                // an error condition exists that should halt data-taking.                             
                // This function should probably make MetricManager calls.                             
                virtual bool checkHWStatus_();

If the “poll_hardware_status” parameter for the fragment generator is set to “true” (it defaults to “false”), then every time before getNext*() is called when the DAQ is in the running state, the user-overridable function checkHWStatus*() is called if it’s determined that it’s been at least N microseconds since the last time checkHWStatus_() was called. Here, the “N” is set via the “hardware_poll_interval_us” parameter, which as of artdaq v1_13_02 defaults to 1000000 (i.e., the hardware is polled once a second).

If a user wants to periodically call checkHWStatus*() but doesn’t want it to be called on the same thread as getNext*(), then there’s the option of setting the “separate_monitoring_thread” parameter to “true” (it defaults to “false”). So, if a user wrote a fragment generator which implemented the checkHWStatus*() function and wanted the function to be called every 10 ms on a thread separate from the thread calling getNext*(), the following parameters would need to be added to the FHiCL parameterizing the fragment generator:

poll_hardware_status: true
hardware_poll_interval_us: 10000
separate_monitoring_thread: true

IMPORTANT: Note that as of artdaq v1_13_02, the hardware status will halt datataking only if the “separate_data_thread” described in the previous section is set to true!

Developing fragment generators which send zero, or more than one, fragment per event

On experiments, it’s often the case that a fragment generator is developed whose getNext_() function is designed to return either no fragments at all (if it’s being used primarily for configuration rather than datataking purposes) or more than one fragment. There are a couple of things to be aware of in this case:

If your fragment generator doesn’t send fragments, you’ll want to be sure to set “separate_data_thread: false” and “request_mode: ignored” in the FHiCL used to configure - see below for details on what “request_mode” is.
You’ll want to add the line “generated_fragments_per_event: ” to your FHiCL, where “” is to be replaced by an integer denoting the number of fragments your fragment generator produces per event.

For reference, here are the relevant parameters that need to be to set in the daq.fragment_receiver block of a BoardReader configuration file in these two cases:

for a BoardReader that doesn’t generate any fragments
- request_mode: ignored
- separate_data_thread: false
- generated_fragments_per_event: 0

for a BoardReader that generates more than one fragment per event
- generated_fragments_per_event:
- fragment_ids: [, .. ]

Built-in support for requesting specific events

Overview

A nontrivial but highly powerful feature is the ability to specify that a fragment generator only send its fragments downstream when it receives a request for data. To activate this feature, no code needs to be added to individual fragment generators as the functionality is built-in to the artdaq::CommandableFragmentGenerator base class they inherit; all that’s needed is to set a few FHiCL parameters, described below. The mechanism through which this requesting occurs involves a dedicated thread which runs in the fragment generator and which listens for “RequestPacket” request objects over a socket using UDP; each request object simply carries with it the sequence ID of the event it’s requesting, a timestamp, and a header checksum (RequestPacket is defined in https://github.com/art-daq/artdaq/repository/revisions/develop/entry/artdaq/DAQrate/detail/RequestMessage.hh ). In the documentation below, when a fragment generator is described as “receiving a request”, what’s technically meant is that the fragment generator has received a request in the form of an instance of RequestPacket via its socket.

The fragment generator keeps track of what the next sequence ID it will assign to the next fragment it sends is - the value for which we’ll call the “event counter” - and when it sends that fragment, it automatically increments this event counter by 1. This is distinct from non-request running, in which it is up to the writer of the fragment generator to manually set the sequence ID in the getNext*() function, rather than to rely on this built-in functionality. The significance of the event counter is the following: the fragment generator will only send a fragment with a payload downstream if the sequence ID of the request it’s considering matches up to the current value of the event counter. If the request’s sequence ID is less than the event counter, the request is considered “obsolete”, and completely ignored. If the request’s sequence ID is greater than the event counter, the fragment generator will take that to mean that it missed an earlier request whose sequence ID would have matched the current event counter. When that happens, the fragment generator sends an “empty” - i.e., payload-free - fragment, stamped with the current event counter, and then it increments the event counter. For all “request modes”, the fragment generator will hold the fragments it’s received via calls to getNext*() in memory until a request is received, at which point it will make mode-dependent decisions as to which fragments to send downstream and which to discard; the buffer, however, is configurable for how many timestamps to keep before discarding data. The buffer is also limited by number of fragments and size of data as defined above in the section on running a separate datataking thread; when the buffer is full, read-out of the hardware will pause.

There are two different ways to run in request mode: “window mode” and “single mode”. In window mode, fragments whose artdaq::Fragment header’s timestamp falls within a time window centered around the request’s timestamp will be sent downstream. This, of course, implies that the timestamp needs to be set properly in the artdaq::Fragment objects returned by an experiment’s implementation of the getNext*() function. The requests for data can be sent from the eventbuilders or from another source. In the first case, an eventbuilder sends a request message when it receives the first fragment with a given sequence ID- meaning there needs to be at least one fragment generator not running in a request mode. In single mode, a fragment generator sends only the most recent fragment it’s received from getNext*() when it receives a request. The request mode is set via a parameter called “request_mode” described below; it defaults to “ignored”, in which case the functionality just described is disabled, and data is sent from the fragment generator without the generator waiting on requests.

Details on these features and how to activate them through FHiCL will now be described.

Common Parameters

Regardless of the request mode, certain parameters will need to be set for the receiving of requests to be supported in your fragment generator. Specifically, we need the following:

separate_data_thread: true
request_mode: <mode>

The request_mode variable should be set to “ignored”, “single”, or “window”. Note that unless you’re in “ignored” mode you need to run the fragment generator with a separate data receiving thread (described earlier in the wiki), otherwise an exception will be thrown. Two other parameters used in all trigger modes are request_port and request_address; these have defaults (“3001” and “227.128.12.26” as of artdaq v2_01_00) so the user doesn’t necessarily need to set them explicitly.

A final parameter common to all request modes is stale_request_timeout. Like many of the other parameters already presented, this has a default - a hexadecimal value of 0xFFFFFFFF, or just under 4.3 billion, the maximum value of an unsigned 4-byte integer. If a fragment returned by getNext_() is more than stale_request_timeout ticks old, it will be deleted, and unavailable for downstream sending the next time a request is sent - even if otherwise the fragment would have been sent.

h4. Window Mode Requests

To set window mode requests, use request_mode: window. The basic concept behind window mode triggering is that when a request is received by the fragment generator, only those fragments returned by getNext_() which contain a timestamp within a certain number of ticks of the request’s timestamp get bundled into a “container” artdaq::Fragment object, whose header is assigned the request’s sequence ID and timestamp. This container fragment is then sent downstream. In the event that the fragment generator sees that the next request is for a sequence ID higher than its current tracked sequence ID, it will send an empty fragment downstream with the current sequence ID, where an empty fragment has no payload and a type of value artdaq::Fragment::EmptyFragmentType. “Window” in this context refers to the time window, the number of ticks before and after the request’s timestamp in which fragments are included. To set these values, one uses the request_window_width parameter to define the tick duration of the window, and request_window_offset to define where in the window the timestamp lies - there need not necessarily be an equal number of ticks before and after the request’s timestamp defining the window. So, e.g., if you set

request_window_width: 10
request_window_offset: 3

and the timestamp on the request is the integer “t0”, you’re requesting that all fragments with a timestamp value in their headers which lies from (t0 - 3) to (t0 - 3 + 10) == (t0 + 7 ), inclusive, be bundled into a container fragment and sent downstream. Note that the number of ticks which compose the window is actually one more than the setting of request_window_width- e.g., in this example, the window is composed of 11 ticks. A final parameter to be aware of is request_windows_are_unique; set to “true” by default, this defines whether or not a given fragment returned from getNext_() can appear in at most one container fragment (“true”) or more than one container fragment (“false”). In the “true” case, a fragment which has already been bundled into a prior container fragment will be unavailable for the next container fragment even if it lies with that next container fragment’s time window.

h4. Single Mode Requests

Setting request_mode: single will enable single mode requests. In this mode, when the fragment generator receives a request, it will stamp the header of the most recently received fragment from getNext*() with the requests’s sequence ID and timestamp and then send it downstream. Other fragments which may have been previously returned by calls to getNext*() on the data acquisition thread and stored in the fragment generator’s memory will have been discarded. One thing to be aware of is that as of artdaq v1_13_03, single mode triggering assumes that at most one fragment per event will be returned by calls to getNext*. If getNext*() returns no fragments on a particular call, then an empty fragment (as defined in the “window mode” section above) will be sent downstream.

How to Send Requests

The fragment generators with a request mode activated are, of course, ignorant as to what sends them the requests. On an experiment, e.g., this might be done by a dedicated process- perhaps even another fragment generator. In order to test request features out-of-the-box, however, it’s possible to have EventBuilders themselves send requests. The way in which this is done is quite simple- the send_requests parameter, which defaults to “false” in the eventbuilder’s FHiCL, needs to be set to “true”. When true, once an EventBuilder receives a fragment with a sequence ID it hasn’t yet seen (i.e., the first fragment in a new event), it will send out a request object with that fragment’s sequence ID and timestamp, to be received by request-enabled fragment generators (meaning, at least one non-request-enabled fragment generator needs to be running in order to send out fragments to the eventbuilder to activate its sending of the request). The default values for the request port and address are the same as the default values in the fragment generators; consequently if you override the defaults by explicitly specifying the port and/or address, you’ll want to override them in both places. A final parameter which can be set in a request-sending eventbuilder is request_delay; this value specifies the microseconds - not the same thing as the tick length in timestamps - between when the EventBuilder receives the first fragment in a new event and when it sends out the request. So, e.g., if I add these parameters to an EventBuilder’s FHiCL document:

send_requests: true
request_delay: 10000

…I’m requesting that the EventBuilder send a request to all request-enabled fragment generators 10 ms after receiving the first fragment in an event, from a non-request-enabled fragment generator, and that this request be sent using the default socket port/address values.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to write a fragment generator

How to write a fragment generator

Intro

An overview on fragment generators

What code gets executed during state transitions

A quick overview of the ToyHardwareInterface API

A breakdown of ToySimulator

More features

Using separate threads for sending fragments downstream to eventbuilders vs. extraction of data from upstream hardware

An option to monitor hardware

Developing fragment generators which send zero, or more than one, fragment per event

Built-in support for requesting specific events

Overview

Common Parameters

How to Send Requests

Clone this wiki locally