Initial log upload support

[peterbarker]

We need some preliminary, rudimentary log upload support. This will be superseded by something more intelligent later.

Currently the DataFlash logs are written to ~apsync/dflogger/dataflash.

We need to get those logs off the companion computer and onto a server somewhere (heretoforth known as "the cloud")

• We need a route from the companion computer to the cloud
• we need to handle the case where logs are deleted from the CC
• we need to handle the CC being rebooted

Proposal:

• add to web interface:
  • ability to define a "to" server (by hostname or IP address) as a text input box in the web interface on port 8000 (CLOUDSYNC_ADDRESS)
  • add ability to paste in the contents of an identity_file to be used when trying to upload logs
  • add ability to specify a username for synchronising logs up (CLOUDSYNC_USER)
  • add a field to contain a unique identifier for this vehicle
• modify a daemon on the system (MAVProxy?) to run a job once/minute
  • if there is no route to the cloud then exit
  • if the vehicle is armed then exit; if the vehicle ever becomes armed then kill the syncing process
  • if the vehicle becomes
  • generate a timestamped destination directory
  • e.g. dataflash-${VEHICLE_UNIQUE_ID}-$(date '+%Y%m%d%H%M%S)` (CLOUDSYNC_DIR)
  • for each file in the dataflash directory, stat it for modifcations over, say, 15 seconds. If not changed, attempt to sync it to cloud
  • invoke rsync, something like this:
    • rsync -aHz $FILEPATH ${CLOUDSYNC_USER}@${CLOUDSYNC_ADDRESS}:$CLOUDSYNC_DIR
  • if the rsync completes successfully, move the file on APSync to a local $HOME/dflogger/dataflash-archive/$CLOUDSYNC_DIR directory

Doing the files individually has the advantage that we will avoid pushing the same file up multiple times into differently-timestamped directories should the rsync transfer be interrupted by e.g. the CC machine being rebooted. The lack of --partial on the receiver end should mean that we don't end up with partial logs on the server, at the expense of transferring the same data multiple times.

Note that simply running rsync directly on the $HOME/dflogger/dataflash directory will seldom succeed as a file is constantly being written in that directory.

[SamuelDudley]

Outdated Proposal - See my next proposal below

@peterbarker My current work adding this feature can be found here. I’m attempting to follow your proposal but have hit a bit of an issue with the cloudsync_dir. Basically the log is rsync’d from the dflogger directory and I can't find a good way to rsync it to a non-existent target directory without first making the directory on the server. For the time being I’m dumping the logs into the cloudsync_user’s home directory on the server.

I’m using pre shared keys for authentication with the server. It’s quite simple for someone to replicate the server setup if they wanted to manage their own logs, but i'm assuming that the end goal is to have one server used by many users & vehicles.

One idea is as follows (edit - added some details):

One time registration with datalog server

• Users register with the datalog server and provide a user name + password via a web front end
• Once logged in on the datalog server (on the web front end), the user can create a new vehicle for the profile
• The server will assign a unique vehicle ID and its ssh fingerprint

One time setup on the companion computer

This can all be automated from a webform being served by the companion computer

• A ssh key pair is generated ssh-keygen -t ed25519
• The datalog server fingerprint is added to ~/.ssh/known_hosts
• The companion computer makes contact with the datalog server and completes a one time upload of the companion computer ssh public key via a TLS/SSL encrypted web socket. The datalog server user name, password and unique vehicle ID is also provided. These details allow the datalog server to check the user name password pair and then associate the ssh connections using this key with the unique vehicle ID

Normal operation

The companion computer wants to rsync a log

• The companion computer contacts the datalog server with the server user name + unique vehicle ID via a TLS/SSL encrypted web socket
• The datalog server responds over the web socket with a time stamped directory name and a valid time period
  • The datalog server creates the time stamped directory
• The time stamped directory now exists on the datalog server, and we can begin to rsync to it using the ssh pre-shared keys and user name
• The datalog server accepts the connection from that particular ssh key and rsync's the datalog
• Once the rsync is complete, the companion computer creates a local archive folder using the name privided by the datalog server. The local copy of the datalog is then moved to this folder.
• When the token expires, if the companion computer fails to upload a log the empty directory is cleaned up on the datalog server

Accessing the logs on the datalog server

When a user wants to retrieve logs they have sent to the datalog server

• Users navigate to the via the datalog server web front end and login using the user name and password
• User is provided with a list of registered vehicles and can browse the associated logs
• There could be further functions to download / perform analysis on the logs

Making this somewhat secure would be a bit of a challenge, but we could chroot jail a user to the assigned folder for that particular one time upload?
https://linuxconfig.org/how-to-automatically-chroot-jail-selected-ssh-user-logins
https://serverfault.com/questions/287578/trying-to-setup-chrootd-rsync

Just ideas, but let me know your thoughts.

[SamuelDudley]

Current Proposal

@davidbuzz mentioned some really good ideas with regards to making the process easier for users. I'll try and capture those ideas here, but in general:

• Remove the concept of passwords and rely only on ssh keys for all authentication
• Remove the concept of user names and utilise an email address to identify users

One time registration with datalog server + setup of the companion computer

Significantly reduces the user setup effort

user steps:

1. Enter the address of the datalog server or leave the default on the companion computer web UI
2. Enter your email and optionally a vehicle ID on the companion computer web UI
3. On the companion computer web UI a button is pushed to 'register'
4. In the received email a confirmation link is clicked

all steps:

• On the companion computer web UI the user enters the address of the datalog server e.g. www.apsync.cloud, their email and an optional unique vehicle ID
• On the companion computer a button is pushed to 'register' with the server
• A ssh key pair is generated automatically (thanks @davidbuzz 👍 )
• A REST endpoint or websocket on the server is pushed / connected to via a TLS/SSL
• The user email, unique vehicle ID (optional) and public key are sent to the server
• An email is sent to the user from the server asking them to confirm via clicking on a link in the email
• Once the link is confirmed, if a vehicle ID was not provided to the server, one is generated and the server associates that public key with the email + vehicle ID combo.
• The companion computer detects the user has confirmed the email via push or challenge REST request and receives a generic username, unique vehicle id, and port number for the server
• The configuration file on the companion computer is automatically updated and saved for future use

Normal operation

The companion computer wants to rsync a log

user steps:

N/A

all steps:

• The companion computer contacts the datalog server with the user email and unique vehicle id via an encrypted web socket or REST endpoint
• The datalog server responds with a time stamped directory name and a valid time period to begin syncing
• The datalog server creates the time stamped directory for that user + vehicle ID
• The time stamped directory now exists on the datalog server, and we can begin to rsync to it using the ssh pre-shared keys and user name
• The datalog server accepts the connection from that particular ssh key and rsync's the datalog
• Once the rsync is complete, the companion computer creates a local archive folder using the name provided by the datalog server. The local copy of the datalog is then moved to this folder.
• If a log upload has not started when the valid time period expires the empty directory is cleaned up on the datalog server

Accessing the logs on the datalog server

When a user wants to retrieve logs that have been rsync'd with the datalog server

user steps via 'normal' access :

1. Navigate to website
2. Login

user steps via companion computer access :

1. Open the correct screen in the companion computer web UI

all steps:

• Users navigate to datalog server web front end and login using the email address and ssh public key string (a bit painful?)
• Perhaps there is an option to login with a linked google / github / ardupilot.org account to simplify things?
• User is provided with a list of registered vehicles and can browse the associated logs
• There could be further functions to download / perform analysis on the logs
• The web UI on the companion computer is also capable of connecting via a REST backend to display a list of sync'd logs for the associated email address (all vehicles) and ssh public key combo