Log file created by default for uploads and downloads

[i-oden]

Read this before submitting the PR

1. Always create a Draft PR first
2. Go through sections 1-5 below, fill them in and check all the boxes
3. Make sure that the branch is updated; if there's an "Update branch" button at the bottom of the PR, rebase or update branch.
4. When all boxes are checked, information is filled in, and the branch is updated: mark as Ready For Review and tag reviewers (top right)
5. Once there is a submitted review, implement the suggestions (if reasonable, otherwise discuss) and request an new review.

If there is a field which you are unsure about, enter the edit mode of this description or go to the PR template; There are invisible comments providing descriptions which may be of help.

1. Description / Summary

People rarely use the --log-file option. Previously the default was that there was always a log file generated, but we were logging a lot, which resulted in the users (mainly the units) generating a huge amount of logs which were quite big. However, we have now reduced the logging a bit.

Every time there's a support ticket, we usually end up having to ask the user to run the command again, which takes both time and energy for us, but also of course means that we cannot immediately investigate what the issue is and often have to wait for the user to try again.

This PR adds a flag --force-no-log which the user can use if they really do not want to generate a log file. If this flag is not used, the dds by default saves all logs to a file with the format: _.log. The user can also use the --log-file option to specify their own path to the log file, as usual.

The code needed some refactoring before being able to do this, and at first I made this PR: #737
However, while that did technically work when it comes to generating the actual log file as I wanted it to, it did not actually log things to the file. In order to make it work I needed to have it in the __main__.py module. There is a slight bit of code duplication since I need to add code to the dds data get and dds data put commands.

I probably need to look at adding tests as well, but will look at that next week.

2. Jira task / GitHub issue

HMS-2292

3. Type of change - Add label

What type of change(s) does the PR contain? For an explanation of the different options below, enter edit mode of this PR description template.

If you do not want this change to be included in release notes, add the label skip-changelog.

• New feature
  • Breaking --> label: breaking
  • Non-breaking --> label: feature
• Database change --> label: feature or none at all.
  Remember the to include a new migration version, or explain here why it's not needed.
• Bug fix --> label: bug
• Security Alert fix
  • Package update --> label: dependency
    • Major version update
• Documentation --> label can be skipped, will be included in "other changes"
• Workflow --> label: skip-changelog
• Tests only --> label: skip-changelog

4. Additional information

• I have added an entry to the Sprintlog
• This is a PR to the master branch: If checked, read the release instructions
  • I have followed steps 1-8.

5. Actions / Scans

Make sure that the following checks/actions have passed.

• Black

• Prettier

• Pylint

• Yamllint

• Tests

• CodeQL

• Trivy

• Snyk

• TestPyPI

If an action does not pass and you need help with how to solve it, enter edit mode of this PR template or go to the PR template.

[i-oden]

P.s. I will rename this PR before we merge it, I just want to keep track of which one is the working one and which one I made first etc.

[i-oden]

@ScilifelabDataCentre/teamhermes Review this when you have a chance

[rv0lt]

Maybe the only detail that I see. Is that when there is an error, the message asks to check for the JSON log and not this new one; example:

Please verify that the following error log has been generated: /code/DataDelivery_2025-04-25_08-49-38_project_1_upload/logs/dds_failed_delivery.json

[i-oden]

Maybe the only detail that I see. Is that when there is an error, the message asks to check for the JSON log and not this new one; example:

Please verify that the following error log has been generated: /code/DataDelivery_2025-04-25_08-49-38_project_1_upload/logs/dds_failed_delivery.json

Good point!

[i-oden]

@rv0lt I added the same info to the download message