-
Notifications
You must be signed in to change notification settings - Fork 2
Description
Environment:
- Delivery tag:
- Platform: OPS Orange Cloud
- Configuration:
S2-L1 1.8.0
Traceability:
Current Behavior:
S2 L1B execution failed with following error "Error during task L1B - Exit code: 1"
Expected Behavior:
Execution should be successfully done.
Steps To Reproduce:
Sample processing S2, error on following execution:
[
{
"filename": "S2A_OPER_MSI_L1A_DS_REFS_20240129T181737_S20240129T113720_N04.00.L1SA"
},
{
"filename": "S2A_OPER_MSI_L1B_DS_REFS_20240129T181737_S20240129T113720_N04.00.L1SA"
}
]
Test execution artefacts (i.e. logs, screenshots…)
pod execution logs:
Explore-logs-2024-01-30 08_18_31.txt
IPF logs Errors:
logs_FAILURE_L1B_T000_20240130085218.tar.gz
Whenever possible, first analysis of the root cause
There was 2 weird behavior.
first one:
Orchestration launcher last execution percent seen at 27% after 489 seconds:
2024-01-29T18:50:31+00:00 {"header":{"type":"LOG","timestamp":"2024-01-29T18:50:31.942363Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-17-thread-1"},"message":{"content":"2024-01-29 18:50:31 [INFO] OrchestratorLauncher : #########################"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
2024-01-29T18:50:31+00:00 {"header":{"type":"LOG","timestamp":"2024-01-29T18:50:31.942080Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-17-thread-1"},"message":{"content":"9"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
2024-01-29T18:50:26+00:00 {"header":{"type":"LOG","timestamp":"2024-01-29T18:50:26.927364Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-17-thread-1"},"message":{"content":"2024-01-29 18:50:26 [INFO] OrchestratorLauncher : Processing L1B : 27% in 489 seconds : 1 / 1 : 14 / 14 : 12 / 24 : 0 / 1 : 0 / 1 : 0 / 1 : 0 / 1 : 0 / 1 : 0 / 1 : "},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
Many step RESAMPLE_TO_VS execution failed:
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227610Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":" 3 RESAMPLE_TO_VS
808082 13 08:59:15 99 280 ERROR 128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227655Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":" 3 RESAMPLE_TO_VS
808084 14 08:59:15 100 281 ERROR 128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227677Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":" 3 RESAMPLE_TO_VS
808088 15 08:59:15 99 281 ERROR 128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227701Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":" 3 RESAMPLE_TO_VS
808090 16 08:59:15 100 281 ERROR 128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227724Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":" 3 RESAMPLE_TO_VS 815809 17 09:00:55 15 281 ERROR 128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227748Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":" 3 RESAMPLE_TO_VS 815811 18 09:00:55 16 281 ERROR 128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227771Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":" 3 RESAMPLE_TO_VS 815859 19 09:00:55 15 273 ERROR 128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227802Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":" 3 RESAMPLE_TO_VS 815905 20 09:00:56 16 270 ERROR 128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227827Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":" 3 RESAMPLE_TO_VS 816705 21 09:01:11 16 255 ERROR 128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227859Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":" 3 RESAMPLE_TO_VS 816706 22 09:01:11 15 255 ERROR 128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227890Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":" 3 RESAMPLE_TO_VS 816755 23 09:01:11 16 281 ERROR 128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227917Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":" 3 RESAMPLE_TO_VS 816801 24 09:01:12 16 276 ERROR 128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227950Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":" 4 TP_COLLECT 0 1 00:00:00.0 0 0 NORUN 666"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227981Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":" 5 TP_FILTER 0 1 00:00:00.0 0 0 NORUN 666"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.228010Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":" 6 SPATIO 0 1 00:00:00.0 0 0 NORUN 666"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.228042Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":" 7 GEO1B_FINALIZE 0 1 00:00:00.0 0 0 NORUN 666"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.228072Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":" 8 FORMAT_METADATA_DS_L1B 0 1 00:00:00.0 0 0 NORUN 666"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.228098Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":" 9 GET_TILE_LIST 0 1 00:00:00.0 0 0 NORUN 666"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
After investigation on IPF log We saw an warning on DEM missing file
2024-01-30T09:00:50.098000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[INFO] com/pds/ipfs/s2ipf/RESAMPLE_TO_VS/functional/cpp/Process.C:process:115: RESAMPLE_TO_VS Process:: Resampling
2024-01-30T09:00:52.549000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[WARNING] com/pds/ipfs/s2ipf/libcommonGMV/functional/algorithms/DEM/cpp/S2_DEMhandler_SRTM.C:setCurrentDtedCell:428: Could not find DEM data file: /dem/S2IPF-DEMSRTM//w017/n10
2024-01-30T09:00:52.674000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[WARNING] com/pds/ipfs/s2ipf/libcommonGMV/functional/algorithms/DEM/cpp/S2_DEMhandler_SRTM.C:setCurrentDtedCell:428: Could not find DEM data file: /dem/S2IPF-DEMSRTM//w018/n09
2024-01-30T09:00:52.750000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[WARNING] com/pds/ipfs/s2ipf/libcommonGMV/functional/algorithms/DEM/cpp/S2_DEMhandler_SRTM.C:setCurrentDtedCell:428: Could not find DEM data file: /dem/S2IPF-DEMSRTM//w017/n09
2024-01-30T09:00:52.757000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[WARNING] com/pds/ipfs/s2ipf/libcommonGMV/functional/algorithms/DEM/cpp/S2_DEMhandler_SRTM.C:setCurrentDtedCell:428: Could not find DEM data file: /dem/S2IPF-DEMSRTM//w016/n09
2024-01-30T09:00:52.891000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[WARNING] com/pds/ipfs/s2ipf/libcommonGMV/functional/algorithms/DEM/cpp/S2_DEMhandler_SRTM.C:setCurrentDtedCell:428: Could not find DEM data file: /dem/S2IPF-DEMSRTM//w017/n08
2024-01-30T09:00:52.893000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[WARNING] com/pds/ipfs/s2ipf/libcommonGMV/functional/algorithms/DEM/cpp/S2_DEMhandler_SRTM.C:setCurrentDtedCell:428: Could not find DEM data file: /dem/S2IPF-DEMSRTM//w016/n08
2024-01-30T09:00:52.952000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[WARNING] com/pds/ipfs/s2ipf/libcommonGMV/functional/algorithms/DEM/cpp/S2_DEMhandler_SRTM.C:setCurrentDtedCell:428: Could not find DEM data file: /dem/S2IPF-DEMSRTM//w015/n08
2024-01-30T09:00:53.687000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[INFO] com/pds/ipfs/s2ipf/RESAMPLE_TO_VS/functional/cpp/Process.C:generateVATFGRI:340: Loading GRI_ORBIT: /workspace/ipf_output_L1B_20240130085218/L1B/TaskTable_20240130T085221/RESAMPLE_TO_VS/input/GRI_ORBIT/S2A_OPER_MSI_L1B_DS_EPA__20160615T072312_S20160609T113637_N02.02
2024-01-30T09:00:54.916000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[INFO] com/pds/ipfs/s2ipf/libcommonGMV/functional/filehandler/cpp/GMVFileUtils.C:callProductReportWriter:111: The file /workspace/ipf_output_L1B_20240130085218/L1B/TaskTable_20240130T085221/RESAMPLE_TO_VS/REPORT_808082/report.xml REPORT has been created
2024-01-30T09:00:54.923000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[INFO] com/pds/ipfs/s2ipf/functional/algorithms/cpp/fileUtils.C:validateSchema:1245: XML file: /workspace/ipf_output_L1B_20240130085218/L1B/TaskTable_20240130T085221/RESAMPLE_TO_VS/REPORT_808082/report.xml validated against the schema successfully
Missing file were not found on shared directory.
heraftter error log in ipf
logs RESAMPLE_TO_VS_808082.err
2024-01-30T09:00:54.913000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[ALARM] com/pds/ipfs/s2ipf/RESAMPLE_TO_VS/cpp/main.C:start:248: Standard exception caught: AUX_DATA folder does not contain a valid GIP_VIEDIR.
2024-01-30T09:00:54.913000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[ALARM] com/pds/ipfs/s2ipf/RESAMPLE_TO_VS/cpp/main.C:start:249: Abort processing
Bug Generic Definition of Ready (DoR)
- The affect version in which the bug has been found is mentioned
- The context and environment of the bug is detailed
- The description of the bug is clear and unambiguous
- The procedure (steps) to reproduce the bug is clearly detailed
- The tested User Story / features is linked to the bug if available
- Logs are attached if available
- A data set attached if available
Bug Generic Definition of Done (DoD)
- the modification implemented (the solution to fix the bug) is described in the bug.
- Unit tests & Continuous integration performed - Test results available - Structural Test coverage reported by SONAR
- Code committed in GIT with right tag or Analysis/Trade Off documentation up-to-date in reference-system-documentation repository
- Code is compliant with coding rules (SONAR Report as evidence)
- Acceptance criteria of the related User story are checked and Passed