-
Notifications
You must be signed in to change notification settings - Fork 6
Home
This repository contains the source code and supporting documents associated with a low-cost hardware data logger designed to capture NMEA2000 and NMEA0183 packets and store them to SD card, with WiFi over-the-air transfer to a capture device (expected to be a smartphone or tablet, although upload direct to the cloud is possible). The goal of the project is to make the device for as low a cost a possible, allowing for as many as possible to be fielded for any given cost, but still to provide everything required to capture, upload, process, and submit the data to the IHO international Data Center for Digital Bathymetry. The ultimate goal to encourage and enable anyone who wants to better map their local area to do so at the lowest possible cost, while at the same time standardising the formatting and submission process to smooth the passage of data into the appropriate database.
General documentation can be found below; troubleshooting information is also available.
The capture hardware has six essential components:
- A microcontroller to sequence operations.
- An SD card to store the captured data.
- A CAN bus controller to talk to the NMEA2000 bus.
- A CAN bus transceiver to provide the physical layer interface to the NMEA2000 bus.
- An RS-422 transceiver and opto-isolators to provide the physical layer interface to the NMEA0183 bus.
- A WiFi interface for communication with the logger, and transfer of data.
Multiple implementations of this design are possible. Note, however, that the ESP32 implementation is the current development model and therefore is (a) the most developed, and (b) the most heavily tested. While backwards compatibility is supported as often as possible, it isn't a primary goal, and therefore non-ESP32 implementations may have less functionality or support for newer modifications.
The firmware used in the logger monitors the input buses, records data in a series of binary format files, and allows interaction from the outside for debugging, inspection, and data off-load purposes.
The firmware is written in C++ using the NMEA2000 library, the appropriate adapter for the hardware implementation, e.g., ESP32 CAN or Arduino Due CAN, and the Arduino support for the microcontroller (e.g., ESP32 Core). Other requirements may be necessary for some implementations (e.g., the Adafruit Bluefruit library for SPI-based Bluetooth connectivity where the microcontroller has no native support).
The firmware typically runs its own WiFi network for data transfer, configuration, and monitoring, and a webserver to allow for interaction. However, it can also be configured to join an infrastructure WiFi network, or even a cellphone hotspot for real-time offload of data. More details here.
The firmware stores the data captured from the bus as a binary data packet on the attached SD card in order to reduce the overhead of storing data. Once transferred, a Python parser script is available to read and reconstitute the data on the command line (and as an import-able library), and can also be used to write or transcribe the binary data files (allowing it to be used to translate other formats into the one used here). The multi-tool script has a sub-command that also allows for translation of files, and WIBL files can be processed using the OpenVBI project tools.
The logger hardware is usually accessed for programming with the firmware, or debugging, through a USB to UART converter, which is expected to have 5V power but 3.3V data (and will break things if it doesn't); for development we typically use this programmer hardware, although a USB-C all in one cable is also available. However, once programmed, it is often simpler to access the logger through the Desktop GUI, which provides capabilities to send arbitrary commands to the logger, set and retrieve configurations, save configurations locally for later reference, set and retrieve "lab defaults" backup configurations, and to inspect and configure advanced facilities (e.g., NMEA0183 sentence filters, JSON metadata, and algorithm requests).
Although still maintained, the Desktop GUI is typically not commonly used currently, since the same functionality (and more) is available via the logger's firmware website. The GUI could be deprecated in future releases.
The concept of operations for the logger is to process the data in the cloud, and to allow for direct submission to the IHO Data Center for Digital Bathymetry. The repository has an implementation using AWS Lambda and associated technologies, and includes a front-end interface for interaction with the system. The setup for which is described here.
In many cases, WIBL loggers will be deployed on smaller boats with limited connectivity, and the expected operational model is for the log files to be off-loaded by WiFi to another device (e.g., a phone, tablet, or laptop) whenever the ship comes into range, after which the mobile device can be used to send the files to the cloud (potentially at a different location). If the host ship has a WiFi infrastructure with Internet access, however, the WIBL logger can be configured to automatically attempt to upload files to a server. The details of the upload server implementation and deployment can be found here.
The distribution has a number of tools packaged as a Python tool with sub-commands. In addition to providing the ability to parse WIBL files to check the contents, they can be used to edit the contents of WIBL binary files (in a limited context), upload files to AWS for processing, and run both the processing (WIBL to GeoJSON) and submission (upload to DCDB) components from the desktop. A data simulator is also provided to generate sample data files for testing. More details here