Pipeline fails if there is no new data in delta folder

[eclipses]

Hi,

In a scenario where there are a lot of tables to be exported having syncrhonization in BC and Pipelines in Synapse scheduled at regular interval can happen that for some tables there will be no new records, that means no new delta.

With the current version the Pipelines fail.

Error Messages:
Operation on target FailIfEntityNotFound failed: Entity with name Customer-18 was found in the deltas.manifest.cdm.json but no directory with that name was found in businesscentralgb/deltas/. This may be the case if no new deltas have been exported.

Operation on target FailIfEntityNotFound failed: Entity with name LSCStore-99001470 was found in the deltas.manifest.cdm.json but no directory with that name was found in businesscentralgb/deltas/. This may be the case if no new deltas have been exported.

Operation on target FailIfEntityNotFound failed: Entity with name GLAccount-15 was found in the deltas.manifest.cdm.json but no directory with that name was found in businesscentralgb/deltas/. This may be the case if no new deltas have been exported.

Operation on target For each entity failed: Activity failed because an inner activity failed; Inner activity name: ConsolidateNewDeltas, Error: Operation on target FailIfEntityNotFound failed: Entity with name GLAccount-15 was found in the deltas.manifest.cdm.json but no directory with that name was found in businesscentralgb/deltas/. This may be the case if no new deltas have been exported.

In my opinion the system should not return a failure but rather an info or warning.

Thank you
Eclipses

[DuttaSoumya]

Hi @eclipses,

The Fail activity performs two functions:

1. It ensures that the pipeline does not fail silently, when no deltas are found. In some scenarios, such as the one you mentioned where no changes took place, this is okay, but in others, it may not.
2. More importantly, however, it ensures that the pipeline exits prematurely if no deltas exist (meaning that it does not need to be run for that entity). If you were to simply remove the Fail activity, the surrounding condition activity would still succeed, and the pipeline would continue to be executed. This will lead to unnecessary dataflow executions, which greatly increases cost. We could circumvent this issue by adding the following activities into the "True" block of the condition activity. However, since the pipeline contains another condition activity later, we cannot do that (we cannot nest activities like that). The way around this would be to move all activities to the right of the "Entity found" condition into a new pipeline and invoke that pipeline from the "True" block of the condition, which also adds another level of complexity.

For this reason, we have elected to leave it as is for the moment. With this information, you are of course free to modify the pipeline as you see fit.

On the other hand, consider monitoring the activities from the Synapse pipeline runs, where you could filter better on the actual error message and ignore those that point to an deltas being absent, if you so wish.

Best regards,
The bc2adls team.

[eclipses]

Hi @DuttaSoumya,

An error in my opinion is misleading as in reality the process/pipeline is not failing as there is not delta to be processed.
The error is not handling properly this case.
I think you could move everything is at the right of the red point to a new IF Condition activity and process only when you have delta files.

Please let me know if it make sense to you

Kind Regards
Eclipses

[tglink72]

Hello,

   We have a client who is looking to move to the newer version of the BC2ADL tool and they also are expressing concerns about the number of false positive errors received in our current testing environment. It tends to complicate identifying actual issues. Eclipses, what has worked for you to make this more efficient?

Thanks

Tom