DTS not sending logs on error/finished update

[PLangowski]

Component

Dasharo Tools Suite

Device

other

Dasharo version

No response

Dasharo Tools Suite version

v2.1.0

Test case ID

No response

Brief summary

After choosing the L in DTS menu, the logs are not sent when an update finishes or if DTS encounters an error.
After an error, the logs can still be sent by launching the shell/rebooting/powering off.

How reproducible

No response

How to reproduce

• Run DTS
• Press L to enable sending logs
• Perform a Dasharo update
• Make the operation fail (e.g. CTRL+C) or finish the update

Expected behavior

DTS logs are sent to cloud.

Actual behavior

DTS logs are not being sent.

Screenshots

No response

Additional context

The problem could be tackled by using trap to make sure logs are sent in case of errors/interruption. However, then the logs would be sent twice (also on exit). We may want to keep this the way it is and update the documentation to inform the users that they should choose one of those options in order to send the logs.

The problem with sending logs after a successful update should be investigated further.

Solutions you've tried

No response

[PLangowski]

We have decided to not send logs automatically on error. We added information to the documentation Dasharo/docs#961.
The second problem still needs to be resolved.

[artur-rs]

@PLangowski @m-iwanicki @DaniilKl, let's discuss the possibilities of including Client information in the filename/other places. This will significantly improve linking the logs with the issues raised by the email/gh-issue/Matrix-channel.

[miczyg1]

From documentation:

The logs will be sent automatically after exiting from the menu (entering shell, powering off the system or rebooting using the options in DTS menu).

I don't think we can leave it like this. What if a sudden power loss happens, or someone reboots with hope the upload succeeds, but in fact it will fail for some reason? In such case someone can end up without a backup if booted over network. I don't think current approach is acceptable. Besides the HCL script still asks whether to upload the results t ocloud, while we have a key shortcut in the main menu for uploading logs. Confusion is significant. Also not sure which consent to send the logs is actually the right one...

[PLangowski]

@miczyg1 As of now, the HCL logs are completely different from DTS logs and are sent separately.
How would you make sure that the logs are sent in case of a sudden power loss?

[m-iwanicki]

Very bad idea is to use

trap update_log DEBUG

and resend logs after each line.

[m-iwanicki]

Maybe we should make sure logs aren't lost between reboots when using DTS via USB (don't store logs in /tmp).
This would probably require:

• some kind of log truncating/rotating so we don't run out of space. Would be good idea to implement it even when saving to /tmp so we don't run out of RAM (not sure if it could happen in usual use cases)
• if system is read-only (e.g. mounted in PiKVM) then maybe create tmpfs overlay so logs can still be written.

@PLangowski I also made draft PR with logging enhancement which should always log stdout, stderr to dts.log and xtrace to dts-verbose.log: Dasharo/dts-scripts#55.

[miczyg1]

@miczyg1 As of now, the HCL logs are completely different from DTS logs and are sent separately. How would you make sure that the logs are sent in case of a sudden power loss?

You can't ensure sending the logs in a case of sudden power loss, but if you den;t send them immediately after the logs are created, the chances of losing them are just increasing. Not even saying how counterintuitive it is that you have to enter a shell or do something else to trigger sending while you asked twice before to send them (one through L shortcut and seconds in the hcl report script)

[m-iwanicki]

Those logs are collected constantly (dts.log (copy of console output), dts-err.log, flashrom.log), we can't send them immediately (unless we want to somehow send log after each script line). And is power loss something that can happen because of DTS? I think that's more of external problem and logs won't be useful for us in that case.

• We could add separate option in UI to send collected logs (won't help if DTS/system crashes) - not sure about this as we already have L to send logs automatically before reboot.
• We could maybe setup some kind of remote logging solution (rsyslog?) - probably more work not only on DTS side

[miczyg1]

I want the HCL logs to be sent immediately after HCL report is done. This does not happen on latest DTS release.

[m-iwanicki]

And I'll repeat, this issue isn't about HCL logs but console logs (stdout, stderr and trace). Unless you want to combine those logs and send them as one? Not sure how to combine it as currently HCL report is send before update starts.

And HCL report logs are send immediately, if it isn't then please create issue. NS51MU initial deployment:

Results of getting data:

Legend:
[OK]             Data get successfully
[UNKNOWN]        Result is unknown
[ERROR]          Error during getting data

Creating archive with logs...
Done! Logs saved to: /root/Notebook_NS50_70MU_1.07.14_201a44bd-a65d-5286-86d0-ab23f86f4666_2024_12_13_11_29_20_765553667.tar.gz
Sending logs to 3mdeb cloud...
Tavinus Cloud Sender v2.2.8

> Using password from environment

SENDING SINGLE FILE
===================

Notebook_NS50_70MU_1.07.14_201a44bd-a65d-5286-86d0-ab23f86f4666_2024_12_13_11_29_20_765553667.tar.gz >
######################################################################### 100.0%

SUMMARY
=======

 > All Curl calls exited without errors and no WebDAV errors were detected
 > Attempt to send completed > Notebook_NS50_70MU_1.07.14_201a44bd-a65d-5286-86d0-ab23f86f4666_2024_12_13_11_29_20_765553667.tar.gz
Thank you for supporting Dasharo!

I checked and logs are available on cloud.