-
Notifications
You must be signed in to change notification settings - Fork 46
Description
Bug description
- The HTCondor options for
stream_outputandstream_errorseem broken due to the way that the output remapping is handled - The handling for similar paths between output and error can be iffy.
Version: v0.1.20
Output streaming
Previously it was possible to have the logfiles at a custom location and also stream to that location.
The output and error arguments now only use the basename of the provided path.
While the remapping approach works when the files are transferred out of the job, it does not work with the file streaming.
The streamed files end up in the same directory as the *.jdl file instead.
The option to stream the logs is great for longer jobs/tasks to check their status, so it would be nice to somehow have this option again.
Example for wrong location of stderr and stdout when streaming:
...
config.custom_content.append(("stream_error", "True"))
config.custom_content.append(("stream_output", "True"))
config.stdout = os.path.join("Output", "Output_$(JobId).txt")
config.stderr = os.path.join("Error", "Error_$(JobId).txt")
...For the purpose of streaming, this leads to an effective path of <htcondor_create_job_file_factory().dir>/Output_$(JobId).txt instead of the expected <htcondor_create_job_file_factory().dir>/Output/Output_$(JobId).txt.
Similar outputs
In addition, the reduction of both output and error to their basename can lead to unexpected issues if the basename of both is the same and only the rest of the provided path differs.
Example for mix up between stderr and stdout:
...
config.stdout = os.path.join("Output", "$(JobId).txt")
config.stderr = os.path.join("Error", "$(JobId).txt")
...leads to
transfer_output_remaps = "$(JobId)$(law_job_postfix).txt = <htcondor_log_directory>/Error/$(JobId)$(law_job_postfix).txt
and there is no entry for Output.