- Adds DATASET_EXTRA_VARIABLES notes

- Updates TESTING to illustrate extra-vars and failure tests

Author: Alan Christie

RULES.md

@@ -95,16 +95,19 @@ Images...
 10. If `DT_DATASET_OUTPUT_FORMAT` is set, the environment variable
     `DT_DATASET_OUTPUT_FILENAME` will be set, i.e. 'data.sdf.gz'
     If the output file ends '.gz' the output must be compressed (using gzip)
-11. **Must** expect the input and output directories to contain pre-existing
+11. The formatter **Must** expect `DT_DATASET_EXTRA_VARIABLES` to be set.
+    The text present is expected to be supported by the formatter having been
+    supplied through the REST API when the Dataset was first uploaded.
+12. **Must** expect the input and output directories to contain pre-existing
     files
-12. **Must not** remove or over-write any files present in the input or
+13. **Must not** remove or over-write any files present in the input or
     output directories
-13. **Should not** expect to be called to perform a file conversion for
+14. **Should not** expect to be called to perform a file conversion for
     a dataset that's already been converted successfully.
-14. **Should** expect to be limited to no more than 1 CPU core ane no
+15. **Should** expect to be limited to no more than 1 CPU core ane no
     more than 1GiB of memory. Importantly, exceeding the memory limit will
     result in the container being terminated
-15. **Must** use a non-zero exit code to indicate an unrecoverable failure
+16. **Must** use a non-zero exit code to indicate an unrecoverable failure
 
 ---
 

TESTING.md

@@ -31,7 +31,19 @@ normally be created by the DataTier Manager, we can run a specific test...
 
 Here we apply our test/success/**1** dataset to the container...
 
+    $ export TEST_TYPE=success
     $ export TEST_DIR=1
     $ export DATASET_FILENAME=dummy.txt
-    $ mkdir -p test/success/${TEST_DIR}/output
+    $ export DATASET_EXTRA_VARIABLES="x=1"
+    $ mkdir -p test/${TEST_TYPE}/${TEST_DIR}/output
+    $ IMAGE_NAME=${PWD##*/} docker-compose up
+
+Or we can run tests based on files in the failure branch of the test tree
+with...
+
+    $ export TEST_TYPE=failure
+    $ export TEST_DIR=1
+    $ export DATASET_FILENAME=dummy.txt
+    $ export DATASET_EXTRA_VARIABLES="x=1"
+    $ mkdir -p test/${TEST_TYPE}/${TEST_DIR}/output
     $ IMAGE_NAME=${PWD##*/} docker-compose up

docker-compose.yaml

@@ -8,6 +8,8 @@
 #
 # - DATASET_FILENAME (default 'dummy.txt')
 # - IMAGE_NAME (default 'formatter')
+# - DATASET_EXTRA_VARIABLES (default '')
+# - TEST_TYPE (default 'success')
 # - TEST_DIR (default '1')
 #
 # See TESTING.md
@@ -20,8 +22,9 @@ services:
     - DT_DATASET_FILENAME=${DATASET_FILENAME:-dummy.txt}
     - DT_DATASET_INPUT_PATH=/dataset/input
     - DT_DATASET_OUTPUT_PATH=/dataset/output
+    - DT_DATASET_EXTRA_VARIABLES=${DATASET_EXTRA_VARIABLES}
     volumes:
-    - ${PWD}/test/success/${TEST_DIR:-1}:/dataset
+    - ${PWD}/test/${TEST_TYPE:-success}/${TEST_DIR:-1}:/dataset
     deploy:
       resources:
         limits:

docker-entrypoint.sh

@@ -6,21 +6,30 @@
 # Refer to RULES.md for comprehensive format support rules,
 # summarised below...
 #
+#    Formatting rules
+#    ----------------
 # 1. The environment variable `DT_DATASET_FILENAME` will be set to
 #    a string value representing the filename of the dataset.
-#    i.e. 'data.sdf.gz'. If compressed it wil end with '.gz'.
+#    i.e. 'data.sdf'. The formatters will not be presented with compressed
+#    files.
 # 2. The environment variable `DT_DATASET_INPUT_PATH` will be
 #    a directory you can find the file defined in part 1
 # 3. The environment variable `DT_DATASET_OUTPUT_PATH` will be
 #    a directory you can write to
 # 4. The output path may not be empty, and the image must be aware of this
 #    possibility. The container must not remove any files (from the input or
 #    or output directory).
+# 5. If available, additional format variables will be presented, as text,
+#    in the variable `DT_DATASET_EXTRA_VARIABLES`.
 #
-# 5. The optional environment variable `DT_DATASET_OUTPUT_FORMAT` will be set
+#    Conversion
+#    ----------
+#    As well as the above, when called upon to change file formats...
+#
+# 1. The optional environment variable `DT_DATASET_OUTPUT_FORMAT` will be set
 #    if a file format conversion, rather than data loading, is to be performed.
 #    This is expected be a MIME type supported by the image
-# 6. If `DT_DATASET_OUTPUT_FORMAT` is set, the environment variable
+# 2. If `DT_DATASET_OUTPUT_FORMAT` is set, the environment variable
 #    `DT_DATASET_OUTPUT_FILENAME` will be set, i.e. 'data.sdf.gz'.
 #    The conversion process simply checks whether the filename ends '.gz'
 #    to determine whether compression (gzip) is required. The file must be

test/failure/1/input/dummy.txt

@@ -0,0 +1 @@
+A dummy test file