Skip to content

[BUG] [OPS] S2 S2-L1B execution failed "Error during task L1B - Exit code: 1" due to missing DEM data file #1087

@suberti-ads

Description

@suberti-ads

Environment:

  • Delivery tag:
  • Platform: OPS Orange Cloud
  • Configuration:
    S2-L1 1.8.0

Traceability:

Current Behavior:
S2 L1B execution failed with following error "Error during task L1B - Exit code: 1"

Expected Behavior:
Execution should be successfully done.

Steps To Reproduce:
Sample processing S2, error on following execution:

[
 {
  "filename": "S2A_OPER_MSI_L1A_DS_REFS_20240129T181737_S20240129T113720_N04.00.L1SA"
 },
 {
  "filename": "S2A_OPER_MSI_L1B_DS_REFS_20240129T181737_S20240129T113720_N04.00.L1SA"
 }
]

Test execution artefacts (i.e. logs, screenshots…)
pod execution logs:
Explore-logs-2024-01-30 08_18_31.txt
IPF logs Errors:
logs_FAILURE_L1B_T000_20240130085218.tar.gz

Whenever possible, first analysis of the root cause

There was 2 weird behavior.
first one:
Orchestration launcher last execution percent seen at 27% after 489 seconds:

2024-01-29T18:50:31+00:00	{"header":{"type":"LOG","timestamp":"2024-01-29T18:50:31.942363Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-17-thread-1"},"message":{"content":"2024-01-29 18:50:31 [INFO] OrchestratorLauncher : #########################"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
2024-01-29T18:50:31+00:00	{"header":{"type":"LOG","timestamp":"2024-01-29T18:50:31.942080Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-17-thread-1"},"message":{"content":"9"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
2024-01-29T18:50:26+00:00	{"header":{"type":"LOG","timestamp":"2024-01-29T18:50:26.927364Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-17-thread-1"},"message":{"content":"2024-01-29 18:50:26 [INFO] OrchestratorLauncher : Processing L1B  : 27% in 489 seconds : 1 / 1 : 14 / 14 : 12 / 24 : 0 / 1 : 0 / 1 : 0 / 1 : 0 / 1 : 0 / 1 : 0 / 1 : "},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}

Many step RESAMPLE_TO_VS execution failed:


{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227610Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     3  RESAMPLE_TO_VS 
        808082         13  08:59:15          99    280  ERROR             128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227655Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     3  RESAMPLE_TO_VS 
        808084         14  08:59:15         100    281  ERROR             128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227677Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     3  RESAMPLE_TO_VS 
        808088         15  08:59:15          99    281  ERROR             128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227701Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     3  RESAMPLE_TO_VS 
        808090         16  08:59:15         100    281  ERROR             128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227724Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     3  RESAMPLE_TO_VS          815809         17  09:00:55          15    281  ERROR             128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227748Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     3  RESAMPLE_TO_VS          815811         18  09:00:55          16    281  ERROR             128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227771Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     3  RESAMPLE_TO_VS          815859         19  09:00:55          15    273  ERROR             128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227802Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     3  RESAMPLE_TO_VS          815905         20  09:00:56          16    270  ERROR             128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227827Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     3  RESAMPLE_TO_VS          816705         21  09:01:11          16    255  ERROR             128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227859Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     3  RESAMPLE_TO_VS          816706         22  09:01:11          15    255  ERROR             128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227890Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     3  RESAMPLE_TO_VS          816755         23  09:01:11          16    281  ERROR             128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227917Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     3  RESAMPLE_TO_VS          816801         24  09:01:12          16    276  ERROR             128"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227950Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     4  TP_COLLECT                   0          1  00:00:00.0         0      0  NORUN             666"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.227981Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     5  TP_FILTER                    0          1  00:00:00.0         0      0  NORUN             666"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.228010Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     6  SPATIO                       0          1  00:00:00.0         0      0  NORUN             666"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.228042Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     7  GEO1B_FINALIZE               0          1  00:00:00.0         0      0  NORUN             666"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.228072Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     8  FORMAT_METADATA_DS_L1B       0          1  00:00:00.0         0      0  NORUN             666"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}
{"header":{"type":"LOG","timestamp":"2024-01-30T09:02:41.228098Z","level":"INFO","line":133,"file":"Iterator.java","thread":"pool-25-thread-1"},"message":{"content":"     9  GET_TILE_LIST                0          1  00:00:00.0         0      0  NORUN             666"},"custom":{"logger_string":"eu.csgroup.coprs.ps2.core.common.utils.ScriptLogger"}}

After investigation on IPF log We saw an warning on DEM missing file


2024-01-30T09:00:50.098000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[INFO] com/pds/ipfs/s2ipf/RESAMPLE_TO_VS/functional/cpp/Process.C:process:115: RESAMPLE_TO_VS Process:: Resampling
2024-01-30T09:00:52.549000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[WARNING] com/pds/ipfs/s2ipf/libcommonGMV/functional/algorithms/DEM/cpp/S2_DEMhandler_SRTM.C:setCurrentDtedCell:428: Could not find DEM data file: /dem/S2IPF-DEMSRTM//w017/n10
2024-01-30T09:00:52.674000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[WARNING] com/pds/ipfs/s2ipf/libcommonGMV/functional/algorithms/DEM/cpp/S2_DEMhandler_SRTM.C:setCurrentDtedCell:428: Could not find DEM data file: /dem/S2IPF-DEMSRTM//w018/n09
2024-01-30T09:00:52.750000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[WARNING] com/pds/ipfs/s2ipf/libcommonGMV/functional/algorithms/DEM/cpp/S2_DEMhandler_SRTM.C:setCurrentDtedCell:428: Could not find DEM data file: /dem/S2IPF-DEMSRTM//w017/n09
2024-01-30T09:00:52.757000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[WARNING] com/pds/ipfs/s2ipf/libcommonGMV/functional/algorithms/DEM/cpp/S2_DEMhandler_SRTM.C:setCurrentDtedCell:428: Could not find DEM data file: /dem/S2IPF-DEMSRTM//w016/n09
2024-01-30T09:00:52.891000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[WARNING] com/pds/ipfs/s2ipf/libcommonGMV/functional/algorithms/DEM/cpp/S2_DEMhandler_SRTM.C:setCurrentDtedCell:428: Could not find DEM data file: /dem/S2IPF-DEMSRTM//w017/n08
2024-01-30T09:00:52.893000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[WARNING] com/pds/ipfs/s2ipf/libcommonGMV/functional/algorithms/DEM/cpp/S2_DEMhandler_SRTM.C:setCurrentDtedCell:428: Could not find DEM data file: /dem/S2IPF-DEMSRTM//w016/n08
2024-01-30T09:00:52.952000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[WARNING] com/pds/ipfs/s2ipf/libcommonGMV/functional/algorithms/DEM/cpp/S2_DEMhandler_SRTM.C:setCurrentDtedCell:428: Could not find DEM data file: /dem/S2IPF-DEMSRTM//w015/n08
2024-01-30T09:00:53.687000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[INFO] com/pds/ipfs/s2ipf/RESAMPLE_TO_VS/functional/cpp/Process.C:generateVATFGRI:340: Loading GRI_ORBIT: /workspace/ipf_output_L1B_20240130085218/L1B/TaskTable_20240130T085221/RESAMPLE_TO_VS/input/GRI_ORBIT/S2A_OPER_MSI_L1B_DS_EPA__20160615T072312_S20160609T113637_N02.02
2024-01-30T09:00:54.916000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[INFO] com/pds/ipfs/s2ipf/libcommonGMV/functional/filehandler/cpp/GMVFileUtils.C:callProductReportWriter:111: The file /workspace/ipf_output_L1B_20240130085218/L1B/TaskTable_20240130T085221/RESAMPLE_TO_VS/REPORT_808082/report.xml REPORT has been created
2024-01-30T09:00:54.923000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[INFO] com/pds/ipfs/s2ipf/functional/algorithms/cpp/fileUtils.C:validateSchema:1245: XML file: /workspace/ipf_output_L1B_20240130085218/L1B/TaskTable_20240130T085221/RESAMPLE_TO_VS/REPORT_808082/report.xml validated against the schema successfully

Missing file were not found on shared directory.

heraftter error log in ipf
logs RESAMPLE_TO_VS_808082.err


2024-01-30T09:00:54.913000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[ALARM] com/pds/ipfs/s2ipf/RESAMPLE_TO_VS/cpp/main.C:start:248: Standard exception caught: AUX_DATA folder does not contain a valid GIP_VIEDIR.
2024-01-30T09:00:54.913000 s2-l1-part1-ew-l1sb-v25-75d75c6c66-wtsft RESAMPLE_TO_VS 06.01.01 [000000815148]:[ALARM] com/pds/ipfs/s2ipf/RESAMPLE_TO_VS/cpp/main.C:start:249: Abort processing

Bug Generic Definition of Ready (DoR)

  • The affect version in which the bug has been found is mentioned
  • The context and environment of the bug is detailed
  • The description of the bug is clear and unambiguous
  • The procedure (steps) to reproduce the bug is clearly detailed
  • The tested User Story / features is linked to the bug if available
  • Logs are attached if available
  • A data set attached if available

Bug Generic Definition of Done (DoD)

  • the modification implemented (the solution to fix the bug) is described in the bug.
  • Unit tests & Continuous integration performed - Test results available - Structural Test coverage reported by SONAR
  • Code committed in GIT with right tag or Analysis/Trade Off documentation up-to-date in reference-system-documentation repository
  • Code is compliant with coding rules (SONAR Report as evidence)
  • Acceptance criteria of the related User story are checked and Passed

Metadata

Metadata

Assignees

No one assigned

    Labels

    CCBIssue for CCBbugSomething isn't workingopsTicket from ADS operation team

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions