Output Messages

[EliottPaillard]

Hello @costas80,

We have started to integrate output messages in our tests. However we would want to display multiples messages for a single testcase.

Do you think it would be possible to implement that one day?

[costas80]

Hi @EliottPaillard,

I have my doubts about this one. The goal of the output section is to provide a concise and user-friendly short message to the user to inform her of the overall result. I'm afraid if we add support for multiple messages this could be abused by e.g. listing a long set of errors as individual messages. Keep in mind also that this message figures also in PDF reports and tooltips, meaning that it should not be something too extensive. There is also a technical challenge here, given that the output conditions are applied in sequence and will stop when a condition match is successful. This allows you to check in sequence for conditions and stop on the most accurate match, avoiding multiple incorrect matches if you would continue to check all conditions.

If you need to create a more complete report with custom outputs for the user you could potentially find useful the DisplayProcessor. This is a processing handler that allows you to pass any set of inputs to create a customised report. You could for example construct this at the end of a test session to show multiple sets of feedback (although this would not be executed if you've set the test case to stop on errors).

To better understand your needs could you explain how you envisage using multiple output messages?

[costas80]

Hi @EliottPaillard. I'm coming back to the feature request you made to allow multiple messages at the end of a test session. A way we could support this would be through a new match attribute in the output section (applied separately for e.g. success and failure messages), with the following supported values:

• first (the default), meaning that the first (sequentially) condition to match will be considered. If none match the default applies.
• all, meaning that all conditions with passing conditions will be considered. If none match the default applies.
• cascade, meaning that once a first condition is found to pass, the check will cascade to subsequent conditions and include them if they also pass. If a subsequent condition check fails then the cascade stops even if later conditions would also pass. If none match the default applies.

For example:

    <output>
        <failure match="all">
            <case>
                <cond>$flag1</cond>
                <message>"The integrity check on your data failed."</message>
            </case>
            <case>
                <cond>$flag2</cond>
                <message>"The syntax check on your data failed."</message>
            </case>
            <default>"Your data failed to be processed correctly. Please check the session log to determine the cause of the failure."</default>
        </failure>
    </output>

In the above example given that match is set to all, both messages could be displayed if both flag1 and flag2 are true. If match is set to first then only the first message would be generated. Finally, the cascade option allows you to fine-tune the display of multiple related messages, without forcing an all or nothing approach.

In terms of how multiple messages are handled:

• On the UI and in PDF reports, you see one message box with multiple lines (line per message).
• In XML reports, the message element would go from being 0..1 to 0..*.

Would the above feature cover what you had in mind?

[EliottPaillard]

That is perfect!
Thank you @costas80

[costas80]

Hi @EliottPaillard. I reopened this as the discussed feature is not yet available. Once it's published on our nightly channel I will post a follow up comment here (with any usage notes if different from what we discussed before), and finally close the issue.

[costas80]

The feature, implemented exactly as discussed, is now available to use through the latest nightly build. A test session producing multiple output messages will display them in a single message box as separate lines (on the UI and in PDF reports):

In the case of XML reports, the relevant message element goes from being 0..1, to being 0..*:

   ...
   <result>SUCCESS</result>
   <message>Output message 1.</message>
   <message>Output message 2.</message>
   <message>Output message 3.</message>
   ...

I'll proceed to close the issue but feel free to reopen if you have any questions on this.