Fix various issues, improve log messages, update documentation, and add regression tests

[davidallendj]

Pull Request Template

Thank you for your contribution! Please ensure the following before submitting:

Checklist

• My code follows the style guidelines of this project
• I have added/updated comments where needed
• I have added tests that prove my fix is effective or my feature works
• I have run make test (or equivalent) locally and all tests pass
• DCO Sign-off: All commits are signed off (git commit -s) with my real name and email
• REUSE Compliance:
  • Each new/modified source file has SPDX copyright and license headers
  • Any non-commentable files include a <filename>.license sidecar
  • All referenced licenses are present in the LICENSES/ directory

Description

This PR adds documentation that about using --insecure with scan after changing the default value to false. It also adds more log messages to find these related issues and fixes an issue related to closing the secrets file too early, and adds regression tests.

Additionally, this adds SerialConsole and CommandShell to the output of the crawler.

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update

---

For more info, see Contributing Guidelines.

[davidallendj]

Still need to add the regression tests before merging.

[cjh1]

Looks good

[cjh1]

Ther is also support for websocket console in RF, the remote console service currently doesn't support it, but we might want to add it here for completeness, not sure if gofish supports it?