From 65e0b496cafd063c0f080fc48747fda1a39b1ba7 Mon Sep 17 00:00:00 2001 From: Duke Date: Thu, 20 Oct 2022 20:52:20 -0700 Subject: [PATCH 01/16] W-11939773-scatterGatherEdits-duke --- .../ROOT/pages/scatter-gather-concept.adoc | 75 ++++++++++--------- 1 file changed, 40 insertions(+), 35 deletions(-) diff --git a/modules/ROOT/pages/scatter-gather-concept.adoc b/modules/ROOT/pages/scatter-gather-concept.adoc index bfa632d4de..f465e047a2 100644 --- a/modules/ROOT/pages/scatter-gather-concept.adoc +++ b/modules/ROOT/pages/scatter-gather-concept.adoc @@ -4,11 +4,12 @@ include::_attributes.adoc[] endif::[] :page-aliases: scatter-gather-xml-reference.adoc -The Scatter-Gather component is a routing event processor that processes a Mule event through different parallel processing routes that contain different event processors. Each route receives a reference to the Mule event and executes a sequence of one or more event processors. Each of these routes uses a separate thread to execute the event processors, and the resulting Mule event can be either the same Mule event without modifications or a new Mule event with its own payload, attributes, and variables. The Scatter-Gather component then combines the Mule events returned by each processing route into a new Mule event that is passed to the next event processor only after every route completes successfully. +The Scatter-Gather component (``) is a routing event processor that processes a Mule event through different parallel processing routes that contain different event processors, which are connector operations or other Mule components. Each route receives a reference to the Mule event and executes a sequence of one or more event processors. Each of these routes uses a separate thread to execute the event processors, and the resulting Mule event can be either the same Mule event without modifications or a new Mule event with its own payload, attributes, and variables. The Scatter-Gather component then combines the Mule events returned by each processing route into a new Mule event that is passed to the next event processor only after every route completes successfully. -The Scatter-Gather component executes each route in parallel, not sequentially. Parallel execution of routes can greatly increase the efficiency of your Mule application and may provide more information than sequential processing. +The Scatter-Gather component executes each route in parallel, not sequentially. Parallel execution of routes can greatly increase the efficiency of your Mule application and can provide more information than sequential processing. -The Scatter-Gather component works with repeatable streams. It does not process nonrepeatable streams, which can be read only once before they are lost. By default, all streams are repeatable in Mule unless a component's streaming strategy is configured to be nonrepeatable. +//TODO: clarify seemingly conflicting info: "It does not process nonrepeatable streams" vs. can be "configured to be nonrepeatable" +The Scatter-Gather component works with repeatable streams by default. It does not process nonrepeatable streams, which can be read only once before they are lost. By default, all streams are repeatable in Mule unless a component's streaming strategy is configured to be nonrepeatable. The following diagram details the behavior of the Scatter-Gather component: @@ -23,24 +24,24 @@ WARNING: Configure your Scatter-Gather component to have at least two routes; ot == Variable Propagation -Every route starts with the same initial variable values. Modifications to a variable within a specific route do not affect other routes. So, if a variable is added or modified in one route, then, after aggregation, the value is defined by that route. If a variable is added or modified by more than one route, the value is added to a list of all the values defined for that variable within all the routes, for example: +Every route starts with the same initial variable values. Modifications to a variable within a specific route do not affect other routes. So, if a variable is added or modified in one route, then, after aggregation, the value is defined by that route. If a variable is added or modified by more than one route, the value is added to an array of all the values defined for that variable within all the routes, for example: [source,xml,linenums] ---- - - + + - - - - - - + + + + + + - + ---- @@ -50,17 +51,19 @@ After aggregation, the variables are: == Error Handling Inside Scatter-Gather Routes -You can use a Try scope in each route of a Scatter-Gather component to handle any errors that might be generated by a route’s event processors. After every route executes, if any has failed with an error, then the Scatter-Gather component throws an error of type `MULE:COMPOSITE_ROUTING`, and event processing does not proceed past the Scatter-Gather component in the flow. Instead, the flow branches to your error-handling event processors. +You can use a Try scope in each route of a Scatter-Gather component to handle any errors generated by a route’s event processors. After every route executes, if any failed with an error, the Scatter-Gather component throws an error of type `MULE:COMPOSITE_ROUTING`, and event processing does not proceed past the Scatter-Gather component in the flow. Instead, the flow branches to your error-handling event processors. -Because the `MULE:COMPOSITE_ROUTING` error object gathers not only errors from routes that failed but also Mule events from successfully completed routes, your application can use the error-handling event processors to process Mule events from the routes that completed +Because the `MULE:COMPOSITE_ROUTING` error object gathers not only errors from routes that failed but also Mule events from successfully completed routes, your application can use the error-handling event processors to process Mule events from the routes that completed. To illustrate how this works, consider the following two cases: -* The routes in a Scatter-Gather component each contain a Try scope. + -One of the routes generates an error that is successfully handled by that route’s Try scope through an `on-error-continue` error handler, so the route is completed successfully. The Scatter-Gather component consolidates the Mule events from all routes into a new Mule event and passes the consolidated event to the next event processor. +* The routes in a Scatter-Gather component each contain a Try scope. ++ +One of the routes generates an error that is successfully handled by that route’s Try scope through an `on-error-continue` error handler, so the route completes successfully. The Scatter-Gather component consolidates the Mule events from all routes into a new Mule event and passes the consolidated event to the next event processor. -* One of the routes in a Scatter-Gather component does not contain a Try scope or contains a Try scope with an error handler that cannot handle the error type, or the error handler is an `on-error-propagate` type. + +* One of the routes in a Scatter-Gather component does not contain a Try scope or contains a Try scope with an error handler that cannot handle the error type, or the error handler is an `on-error-propagate` type: ++ An error occurs in this route, causing the route to fail, which in turn causes the Scatter-Gather component to throw a `MULE:COMPOSITE_ROUTING` error. The flow branches to your error-handling event processors, which are able to process the Mule events from the completed routes. Example of handling these errors: @@ -76,7 +79,7 @@ Example of handling these errors: - + @@ -93,12 +96,12 @@ Example of handling these errors: == Handle Timeout Errors in a Scatter-Gather -If you configure a timeout for a Scatter-Gather component and a route does not complete processing before the timeout expires, the route throws a `MULE:TIMEOUT` error. This error is then handled the same way as any other error generated from a route: after each route completes (either by processing success or by throwing a `MULE:TIMEOUT` error), the successful results and errors are collected together in the Scatter-Gather component `MULE:COMPOSITE_ROUTING` error, which is then processed in your configured error handler. +If you configure a timeout for a Scatter-Gather component and a route does not complete processing before the timeout expires, the route throws a `MULE:TIMEOUT` error. This error is then handled the same way as any other error generated from a route: after each route completes (either by processing successfully or by throwing a `MULE:TIMEOUT` error), the successful results and errors are collected into the Scatter-Gather component `MULE:COMPOSITE_ROUTING` error, which is then processed in your configured error handler. == Example Project -In Anypoint Studio, you can download and open the example project _Scatter-Gather Flow Control_ from Anypoint Exchange to learn more about how to use the Scatter-Gather component. This example shows the usage of the scatter-gather control flow to aggregate data in parallel and return the result in JSON. +In Anypoint Studio, you can download and open the example project https://www.mulesoft.com/exchange/org.mule.examples/scatter-gather-flow-control/[+Scatter-Gather Flow Control] from Anypoint Exchange to learn more about how to use the Scatter-Gather component. This example shows the use of the scatter-gather control flow to aggregate data in parallel and return the result in JSON. -The example uses prepared data as input for two resources that should be aggregated. The data represents information about two contacts and has the following structure: +The example uses prepared data as input for two resources to be aggregated. The data represents information about two contacts and has the following structure: |=== |Resource|`firstname`|`surname`|`phone`|`email` @@ -115,28 +118,29 @@ The example uses prepared data as input for two resources that should be aggrega |jane.doe@texasComp.com |=== -DataWeave is used to aggregate the data. The information about the contacts is aggregated to a JSON structure that represents data from both resources. +The example uses DataWeave to aggregate the data. The contact information is aggregated to a JSON structure that represents data from both resources. -To download and open this example project while you are in Anypoint Studio, click the Exchange icon in the upper-left corner. Then, in the window that opens, log into Anypoint Exchange and search on the name of the project. +To download and open this example project through Anypoint Studio, click the Exchange icon in the upper-left corner. Then, in the window that opens, log into Anypoint Exchange and search on the name of the project. == Scatter-Gather XML Reference [%header%autowidth.spread,cols="a,a"] |=== |Element |Description -| `scatter-gather` |Sends a request message to multiple targets concurrently. It collects the responses from all routes, and aggregates them into a single message. +| `scatter-gather` |Sends a request message to multiple targets concurrently. This scope collects the responses from all routes and aggregates them into a single message. 2+| *Attributes* -|`timeout` |Sets the timeout for responses from sent messages, in milliseconds. A value of 0 or lower than 0 means no timeout. +|`timeout` |Sets the timeout for responses from sent messages, in milliseconds. A value of 0 or less than 0 means no timeout. -| `maxConcurrency` |Determines the maximum amount of concurrent routes to process. + -By default all routes run in parallel. +| `maxConcurrency` |Determines the maximum number of concurrent routes to process. -By setting this value to 1, scatter-gather processes the routes sequentially. +* By default all routes run in parallel. -| `target` | The name of the target variable. +* By setting this value to 1, `scatter-gather` processes the routes sequentially. -| `targetValue` | Value of the data to store in the target variable. + -If not set, the default value is `#[payload]`. + +| `target` | The name of the xref:target-variables.adoc[target variable]. + +| `targetValue` | Value of the data to store in the target variable. +If not set, the default value is `#[payload]`. This field accepts any value that a variable accepts: * Any supported data type. @@ -158,7 +162,8 @@ This field accepts any value that a variable accepts: == See Also -* xref:about-components.adoc[Core Components] +* xref:about-components.adoc[] * xref:transaction-management.adoc#tx_scopes_routers[How Transactions Affect Scopes and Routers] -* xref:error-handling.adoc[Error Handling] -* xref:try-scope-concept.adoc[Try Scope] +* xref:error-handling.adoc[] +* xref:try-scope-concept.adoc[] +* xref:about-mule-event.adoc[] From eac0773d277fad9f05a56b31ce84191b111a464e Mon Sep 17 00:00:00 2001 From: Duke Date: Thu, 20 Oct 2022 20:59:07 -0700 Subject: [PATCH 02/16] Update modules/ROOT/pages/scatter-gather-concept.adoc --- modules/ROOT/pages/scatter-gather-concept.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/pages/scatter-gather-concept.adoc b/modules/ROOT/pages/scatter-gather-concept.adoc index f465e047a2..61fd22ef1c 100644 --- a/modules/ROOT/pages/scatter-gather-concept.adoc +++ b/modules/ROOT/pages/scatter-gather-concept.adoc @@ -79,7 +79,7 @@ Example of handling these errors: - + From f1db137daa3f7eb933e804c128b4288ebb5f41cf Mon Sep 17 00:00:00 2001 From: Duke Date: Thu, 20 Oct 2022 20:59:36 -0700 Subject: [PATCH 03/16] Update modules/ROOT/pages/scatter-gather-concept.adoc --- modules/ROOT/pages/scatter-gather-concept.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/pages/scatter-gather-concept.adoc b/modules/ROOT/pages/scatter-gather-concept.adoc index 61fd22ef1c..ce1df524b1 100644 --- a/modules/ROOT/pages/scatter-gather-concept.adoc +++ b/modules/ROOT/pages/scatter-gather-concept.adoc @@ -99,7 +99,7 @@ Example of handling these errors: If you configure a timeout for a Scatter-Gather component and a route does not complete processing before the timeout expires, the route throws a `MULE:TIMEOUT` error. This error is then handled the same way as any other error generated from a route: after each route completes (either by processing successfully or by throwing a `MULE:TIMEOUT` error), the successful results and errors are collected into the Scatter-Gather component `MULE:COMPOSITE_ROUTING` error, which is then processed in your configured error handler. == Example Project -In Anypoint Studio, you can download and open the example project https://www.mulesoft.com/exchange/org.mule.examples/scatter-gather-flow-control/[+Scatter-Gather Flow Control] from Anypoint Exchange to learn more about how to use the Scatter-Gather component. This example shows the use of the scatter-gather control flow to aggregate data in parallel and return the result in JSON. +In Anypoint Studio, you can download and open the example project https://www.mulesoft.com/exchange/org.mule.examples/scatter-gather-flow-control/[^Scatter-Gather Flow Control] from Anypoint Exchange to learn more about how to use the Scatter-Gather component. This example shows the use of the scatter-gather control flow to aggregate data in parallel and return the result in JSON. The example uses prepared data as input for two resources to be aggregated. The data represents information about two contacts and has the following structure: From 756632b523a3fdb1103b59b0444aa7aa5506bb37 Mon Sep 17 00:00:00 2001 From: Duke Date: Thu, 20 Oct 2022 21:02:07 -0700 Subject: [PATCH 04/16] Update modules/ROOT/pages/scatter-gather-concept.adoc --- modules/ROOT/pages/scatter-gather-concept.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/pages/scatter-gather-concept.adoc b/modules/ROOT/pages/scatter-gather-concept.adoc index ce1df524b1..557efd978b 100644 --- a/modules/ROOT/pages/scatter-gather-concept.adoc +++ b/modules/ROOT/pages/scatter-gather-concept.adoc @@ -99,7 +99,7 @@ Example of handling these errors: If you configure a timeout for a Scatter-Gather component and a route does not complete processing before the timeout expires, the route throws a `MULE:TIMEOUT` error. This error is then handled the same way as any other error generated from a route: after each route completes (either by processing successfully or by throwing a `MULE:TIMEOUT` error), the successful results and errors are collected into the Scatter-Gather component `MULE:COMPOSITE_ROUTING` error, which is then processed in your configured error handler. == Example Project -In Anypoint Studio, you can download and open the example project https://www.mulesoft.com/exchange/org.mule.examples/scatter-gather-flow-control/[^Scatter-Gather Flow Control] from Anypoint Exchange to learn more about how to use the Scatter-Gather component. This example shows the use of the scatter-gather control flow to aggregate data in parallel and return the result in JSON. +In Anypoint Studio, you can download and open the example project https://www.mulesoft.com/exchange/org.mule.examples/scatter-gather-flow-control/[Scatter-Gather Flow Control^] from Anypoint Exchange to learn more about how to use the Scatter-Gather component. This example shows the use of the scatter-gather control flow to aggregate data in parallel and return the result in JSON. The example uses prepared data as input for two resources to be aggregated. The data represents information about two contacts and has the following structure: From 7fe5e53a22325d297a2f31d85bc89b55c8ea1ac9 Mon Sep 17 00:00:00 2001 From: dukesphere Date: Thu, 20 Oct 2022 21:57:30 -0700 Subject: [PATCH 05/16] W-11939773-scatterGatherEdits-duke --- .../ROOT/pages/scatter-gather-concept.adoc | 43 +++++++++++++++---- 1 file changed, 35 insertions(+), 8 deletions(-) diff --git a/modules/ROOT/pages/scatter-gather-concept.adoc b/modules/ROOT/pages/scatter-gather-concept.adoc index 557efd978b..099564faee 100644 --- a/modules/ROOT/pages/scatter-gather-concept.adoc +++ b/modules/ROOT/pages/scatter-gather-concept.adoc @@ -58,11 +58,11 @@ Because the `MULE:COMPOSITE_ROUTING` error object gathers not only errors from r To illustrate how this works, consider the following two cases: -* The routes in a Scatter-Gather component each contain a Try scope. +* The routes in a Scatter-Gather component each contain a Try scope. + One of the routes generates an error that is successfully handled by that route’s Try scope through an `on-error-continue` error handler, so the route completes successfully. The Scatter-Gather component consolidates the Mule events from all routes into a new Mule event and passes the consolidated event to the next event processor. -* One of the routes in a Scatter-Gather component does not contain a Try scope or contains a Try scope with an error handler that cannot handle the error type, or the error handler is an `on-error-propagate` type: +* One of the routes in a Scatter-Gather component does not contain a Try scope or contains a Try scope with an error handler that cannot handle the error type, or the error handler is an `on-error-propagate` type: + An error occurs in this route, causing the route to fail, which in turn causes the Scatter-Gather component to throw a `MULE:COMPOSITE_ROUTING` error. The flow branches to your error-handling event processors, which are able to process the Mule events from the completed routes. @@ -72,7 +72,7 @@ Example of handling these errors: - + @@ -94,6 +94,33 @@ Example of handling these errors: ---- +The Logger output looks like this (edited for readability): + +[source, logs] +---- +WARN ...LoggerMessageProcessor: +org.mule.runtime.core.internal.message.ErrorBuilder$DeserializableErrorImplementation +{ + description=My Error + detailedDescription=My Error + errorType=APP:MYERROR + cause=org.mule.runtime.api.exception.DefaultMuleException + errorMessage=- + suppressedErrors=[] + childErrors=[] +} +WARN ...LoggerMessageProcessor: null +WARN ...LoggerMessageProcessor: null +WARN ...LoggerMessageProcessor: +org.mule.runtime.core.internal.message.DefaultMessageBuilder$MessageImplementation +{ + payload=apple + mediaType=*/* + attributes= + attributesMediaType=*/* +} +---- + == Handle Timeout Errors in a Scatter-Gather If you configure a timeout for a Scatter-Gather component and a route does not complete processing before the timeout expires, the route throws a `MULE:TIMEOUT` error. This error is then handled the same way as any other error generated from a route: after each route completes (either by processing successfully or by throwing a `MULE:TIMEOUT` error), the successful results and errors are collected into the Scatter-Gather component `MULE:COMPOSITE_ROUTING` error, which is then processed in your configured error handler. @@ -109,13 +136,13 @@ The example uses prepared data as input for two resources to be aggregated. The |John |Doe |096548763 -|john.doe@texasComp.com +|+john.doe@texasComp.com+ |contacts-2.csv |Jane |Doe |091558780 -|jane.doe@texasComp.com +|+jane.doe@texasComp.com+ |=== The example uses DataWeave to aggregate the data. The contact information is aggregated to a JSON structure that represents data from both resources. @@ -131,7 +158,7 @@ To download and open this example project through Anypoint Studio, click the Exc 2+| *Attributes* |`timeout` |Sets the timeout for responses from sent messages, in milliseconds. A value of 0 or less than 0 means no timeout. -| `maxConcurrency` |Determines the maximum number of concurrent routes to process. +| `maxConcurrency` |Determines the maximum number of concurrent routes to process. * By default all routes run in parallel. @@ -139,8 +166,8 @@ To download and open this example project through Anypoint Studio, click the Exc | `target` | The name of the xref:target-variables.adoc[target variable]. -| `targetValue` | Value of the data to store in the target variable. -If not set, the default value is `#[payload]`. +| `targetValue` | Value of the data to store in the target variable. +If not set, the default value is `#[payload]`. This field accepts any value that a variable accepts: * Any supported data type. From 06d3c2889ac894b64ae2a1fef5e9d124956a0d56 Mon Sep 17 00:00:00 2001 From: Duke Date: Mon, 31 Oct 2022 15:02:12 -0700 Subject: [PATCH 06/16] Update modules/ROOT/pages/scatter-gather-concept.adoc --- modules/ROOT/pages/scatter-gather-concept.adoc | 1 - 1 file changed, 1 deletion(-) diff --git a/modules/ROOT/pages/scatter-gather-concept.adoc b/modules/ROOT/pages/scatter-gather-concept.adoc index 099564faee..342cbbf54d 100644 --- a/modules/ROOT/pages/scatter-gather-concept.adoc +++ b/modules/ROOT/pages/scatter-gather-concept.adoc @@ -8,7 +8,6 @@ The Scatter-Gather component (``) is a routing event processor The Scatter-Gather component executes each route in parallel, not sequentially. Parallel execution of routes can greatly increase the efficiency of your Mule application and can provide more information than sequential processing. -//TODO: clarify seemingly conflicting info: "It does not process nonrepeatable streams" vs. can be "configured to be nonrepeatable" The Scatter-Gather component works with repeatable streams by default. It does not process nonrepeatable streams, which can be read only once before they are lost. By default, all streams are repeatable in Mule unless a component's streaming strategy is configured to be nonrepeatable. The following diagram details the behavior of the Scatter-Gather component: From 381fc2becf4ae53425d92bef2f8d3cb53ada9ac6 Mon Sep 17 00:00:00 2001 From: dukesphere Date: Mon, 31 Oct 2022 15:02:34 -0700 Subject: [PATCH 07/16] W-11939773-scatterGatherEdits-duke --- modules/ROOT/pages/scatter-gather-concept.adoc | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/modules/ROOT/pages/scatter-gather-concept.adoc b/modules/ROOT/pages/scatter-gather-concept.adoc index 099564faee..e9ee03cdcb 100644 --- a/modules/ROOT/pages/scatter-gather-concept.adoc +++ b/modules/ROOT/pages/scatter-gather-concept.adoc @@ -8,9 +8,6 @@ The Scatter-Gather component (``) is a routing event processor The Scatter-Gather component executes each route in parallel, not sequentially. Parallel execution of routes can greatly increase the efficiency of your Mule application and can provide more information than sequential processing. -//TODO: clarify seemingly conflicting info: "It does not process nonrepeatable streams" vs. can be "configured to be nonrepeatable" -The Scatter-Gather component works with repeatable streams by default. It does not process nonrepeatable streams, which can be read only once before they are lost. By default, all streams are repeatable in Mule unless a component's streaming strategy is configured to be nonrepeatable. - The following diagram details the behavior of the Scatter-Gather component: image::mruntime-scatter-gather.png[Diagram of Scatter-Gather Component] @@ -149,6 +146,14 @@ The example uses DataWeave to aggregate the data. The contact information is agg To download and open this example project through Anypoint Studio, click the Exchange icon in the upper-left corner. Then, in the window that opens, log into Anypoint Exchange and search on the name of the project. +== Repeatable Stream Input + +The Scatter-Gather component can process only repeatable streams. Scatter-Gather cannot process a non-repeatable stream as input from an upstream component. + +Repeatable streams are a Mule feature that enables a component to re-read a stream. Non-repeatable streams can be read only once before they are lost. + +By default, Mule components return a repeatable stream unless their streaming strategy is configured to be non-repeatable. + == Scatter-Gather XML Reference [%header%autowidth.spread,cols="a,a"] From 4b930cacad4c1c69c4e417274816d0f91b538826 Mon Sep 17 00:00:00 2001 From: Duke Date: Mon, 31 Oct 2022 15:26:59 -0700 Subject: [PATCH 08/16] Update modules/ROOT/pages/scatter-gather-concept.adoc --- modules/ROOT/pages/scatter-gather-concept.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/pages/scatter-gather-concept.adoc b/modules/ROOT/pages/scatter-gather-concept.adoc index a70f8823d3..f91daed637 100644 --- a/modules/ROOT/pages/scatter-gather-concept.adoc +++ b/modules/ROOT/pages/scatter-gather-concept.adoc @@ -153,7 +153,7 @@ To download and open this example project through Anypoint Studio, click the Exc == Repeatable Stream Input -The Scatter-Gather component can process only repeatable streams. Scatter-Gather cannot process a non-repeatable stream as input from an upstream component. +The Scatter-Gather component can process only repeatable streams. Scatter-Gather cannot process a non-repeatable stream, which it can receive as input from an upstream component that is configured with a non-repeatable streaming strategy. Repeatable streams are a Mule feature that enables a component to re-read a stream. Non-repeatable streams can be read only once before they are lost. From 2b5490f0b9a527dedaa413108977921e31b8a3fa Mon Sep 17 00:00:00 2001 From: dukesphere Date: Mon, 31 Oct 2022 15:27:45 -0700 Subject: [PATCH 09/16] W-11939773-scatterGatherEdits-duke --- modules/ROOT/pages/scatter-gather-concept.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/ROOT/pages/scatter-gather-concept.adoc b/modules/ROOT/pages/scatter-gather-concept.adoc index f91daed637..061c405eb5 100644 --- a/modules/ROOT/pages/scatter-gather-concept.adoc +++ b/modules/ROOT/pages/scatter-gather-concept.adoc @@ -153,9 +153,9 @@ To download and open this example project through Anypoint Studio, click the Exc == Repeatable Stream Input -The Scatter-Gather component can process only repeatable streams. Scatter-Gather cannot process a non-repeatable stream, which it can receive as input from an upstream component that is configured with a non-repeatable streaming strategy. +The Scatter-Gather component can process only repeatable streams. Scatter-Gather cannot process a non-repeatable stream, which the scope can receive as input from an upstream component that is configured with a non-repeatable streaming strategy. -Repeatable streams are a Mule feature that enables a component to re-read a stream. Non-repeatable streams can be read only once before they are lost. +Repeatable streams are a Mule feature that enables a component to re-read a stream. Non-repeatable streams can be read only once before they are lost. By default, Mule components return a repeatable stream unless their streaming strategy is configured to be non-repeatable. From 5a1771c54417695165ec1a08db3152edc4156e9a Mon Sep 17 00:00:00 2001 From: Duke Date: Mon, 31 Oct 2022 17:14:46 -0700 Subject: [PATCH 10/16] W-11939773-scatterGatherEdits-duke fix merge conflict --- modules/ROOT/pages/scatter-gather-concept.adoc | 5 ----- 1 file changed, 5 deletions(-) diff --git a/modules/ROOT/pages/scatter-gather-concept.adoc b/modules/ROOT/pages/scatter-gather-concept.adoc index 061c405eb5..0f0294d419 100644 --- a/modules/ROOT/pages/scatter-gather-concept.adoc +++ b/modules/ROOT/pages/scatter-gather-concept.adoc @@ -8,11 +8,6 @@ The Scatter-Gather component (``) is a routing event processor The Scatter-Gather component executes each route in parallel, not sequentially. Parallel execution of routes can greatly increase the efficiency of your Mule application and can provide more information than sequential processing. -<<<<<<< HEAD -======= -The Scatter-Gather component works with repeatable streams by default. It does not process nonrepeatable streams, which can be read only once before they are lost. By default, all streams are repeatable in Mule unless a component's streaming strategy is configured to be nonrepeatable. - ->>>>>>> 06d3c2889ac894b64ae2a1fef5e9d124956a0d56 The following diagram details the behavior of the Scatter-Gather component: image::mruntime-scatter-gather.png[Diagram of Scatter-Gather Component] From 3ca33f3199f3e505d73dc6626a8d8bde367246ed Mon Sep 17 00:00:00 2001 From: dukesphere Date: Mon, 31 Oct 2022 17:56:16 -0700 Subject: [PATCH 11/16] W-11939773-scatterGatherEdits-duke add ex --- .../ROOT/pages/scatter-gather-concept.adoc | 113 +++++++++++++++++- 1 file changed, 111 insertions(+), 2 deletions(-) diff --git a/modules/ROOT/pages/scatter-gather-concept.adoc b/modules/ROOT/pages/scatter-gather-concept.adoc index 0f0294d419..9147da3ea9 100644 --- a/modules/ROOT/pages/scatter-gather-concept.adoc +++ b/modules/ROOT/pages/scatter-gather-concept.adoc @@ -4,7 +4,7 @@ include::_attributes.adoc[] endif::[] :page-aliases: scatter-gather-xml-reference.adoc -The Scatter-Gather component (``) is a routing event processor that processes a Mule event through different parallel processing routes that contain different event processors, which are connector operations or other Mule components. Each route receives a reference to the Mule event and executes a sequence of one or more event processors. Each of these routes uses a separate thread to execute the event processors, and the resulting Mule event can be either the same Mule event without modifications or a new Mule event with its own payload, attributes, and variables. The Scatter-Gather component then combines the Mule events returned by each processing route into a new Mule event that is passed to the next event processor only after every route completes successfully. +The Scatter-Gather component (``) is a routing event processor that processes a Mule event through different parallel processing routes that contain different event processors, which are connector operations or other Mule components. Each route receives a reference to the Mule event and executes a sequence of one or more event processors. Each of these routes uses a separate thread to execute the event processors on the input, and the resulting Mule event can be either the same Mule event without modifications or a new Mule event with its own payload, attributes, and variables. The Scatter-Gather component then combines the Mule events returned by each processing route into a new Mule event object that passes to the next event processor only after every route completes successfully. The Scatter-Gather component executes each route in parallel, not sequentially. Parallel execution of routes can greatly increase the efficiency of your Mule application and can provide more information than sequential processing. @@ -17,7 +17,116 @@ image::mruntime-scatter-gather.png[Diagram of Scatter-Gather Component] . Each of the processing routes starts executing in parallel. After all processors inside a route finish processing, the route returns a Mule event, which can be either the same Mule event without modifications or a new Mule event created by the processors in the route as a result of the modifications applied. . After all processing routes have finished execution, the Scatter-Gather component creates a new Mule event that combines all resulting Mule events from each route, and then passes the new Mule event to the next component in the flow. -WARNING: Configure your Scatter-Gather component to have at least two routes; otherwise, your Mule application throws an exception and does not start. +WARNING: Configure your Scatter-Gather component to have at least two routes. Otherwise, your Mule application throws an exception and does not start. + +== Example + +Assume a scheduled input (`<`) to Scatter Gather with the message `"hello scatter gather"` from Set Payload (``). + +[source, xml] +---- + + + + + + + + > + + + + + + + + + + + + + + + + + +---- + +* The first `` within Scatter-Gather adds the string `" first route"` to the input `payload`. +* The second `` within Scatter-Gather adds the string `" second route"` to the input `payload`. + +The first Logger after Scatter gather prints the output `payload` of Scatter-Gather, which is an objection containing key-value pairs for each route. The key provides the index of the route (0 or 1), and the value provides a set of key-value pairs not only for the payload but for attributes and other properties of the event, excluding Mule variables, for example: + +[source, JSON] +---- +INFO ...LoggerMessageProcessor: +{ + "0": { + "inboundAttachmentNames": [ + + ], + "exceptionPayload": null, + "inboundPropertyNames": [ + + ], + "outboundAttachmentNames": [ + + ], + "payload": "hello scatter gather first route", + "outboundPropertyNames": [ + + ], + "attributes": null + }, + "1": { + "inboundAttachmentNames": [ + + ], + "exceptionPayload": null, + "inboundPropertyNames": [ + + ], + "outboundAttachmentNames": [ + + ], + "payload": "hello scatter gather second route", + "outboundPropertyNames": [ + + ], + "attributes": null + }, + "2": { + "inboundAttachmentNames": [ + + ], + "exceptionPayload": null, + "inboundPropertyNames": [ + + ], + "outboundAttachmentNames": [ + + ], + "payload": "hello scatter gather third route", + "outboundPropertyNames": [ + + ], + "attributes": null + } +} +---- + +To access and manipulate such a payload, you can use a DataWeave function such as `mapObject`. The second Logger in the example uses `valuesOf(payload mapObject { ($$) : $}).payload` to print the payloads of each key in the output object: + +[source, JSON] +---- +INFO ...LoggerMessageProcessor: +[ + "hello scatter gather first route", + "hello scatter gather second route" +] +---- == Variable Propagation From 33eb801261aff22d6a02085555e123c30f61b41e Mon Sep 17 00:00:00 2001 From: dukesphere Date: Mon, 31 Oct 2022 22:22:10 -0700 Subject: [PATCH 12/16] W-11939773-scatterGatherEdits-duke ex --- .../ROOT/pages/scatter-gather-concept.adoc | 32 ++----------------- 1 file changed, 3 insertions(+), 29 deletions(-) diff --git a/modules/ROOT/pages/scatter-gather-concept.adoc b/modules/ROOT/pages/scatter-gather-concept.adoc index 9147da3ea9..0f4476e82b 100644 --- a/modules/ROOT/pages/scatter-gather-concept.adoc +++ b/modules/ROOT/pages/scatter-gather-concept.adoc @@ -19,9 +19,7 @@ image::mruntime-scatter-gather.png[Diagram of Scatter-Gather Component] WARNING: Configure your Scatter-Gather component to have at least two routes. Otherwise, your Mule application throws an exception and does not start. -== Example - -Assume a scheduled input (`<`) to Scatter Gather with the message `"hello scatter gather"` from Set Payload (``). +To understand how scatter-gather processing of a payload occurs, assume that the component receives the message payload `"hello scatter gather"` from Set Payload (``): [source, xml] ---- @@ -55,9 +53,9 @@ Assume a scheduled input (`<`) to Scatter Gather with the message ` ---- * The first `` within Scatter-Gather adds the string `" first route"` to the input `payload`. -* The second `` within Scatter-Gather adds the string `" second route"` to the input `payload`. +* The second `` within Scatter-Gather adds the string `" second route"` to the same input `payload`. -The first Logger after Scatter gather prints the output `payload` of Scatter-Gather, which is an objection containing key-value pairs for each route. The key provides the index of the route (0 or 1), and the value provides a set of key-value pairs not only for the payload but for attributes and other properties of the event, excluding Mule variables, for example: +The first Logger after the component prints the output `payload` of Scatter-Gather, which is an object containing key-value pairs for each route. The key provides the index of the route (0 or 1), and the value provides a set of key-value pairs for the payload, attributes, and other properties of the event, _excluding_ Mule variables: [source, JSON] ---- @@ -231,30 +229,6 @@ org.mule.runtime.core.internal.message.DefaultMessageBuilder$MessageImplementati If you configure a timeout for a Scatter-Gather component and a route does not complete processing before the timeout expires, the route throws a `MULE:TIMEOUT` error. This error is then handled the same way as any other error generated from a route: after each route completes (either by processing successfully or by throwing a `MULE:TIMEOUT` error), the successful results and errors are collected into the Scatter-Gather component `MULE:COMPOSITE_ROUTING` error, which is then processed in your configured error handler. -== Example Project -In Anypoint Studio, you can download and open the example project https://www.mulesoft.com/exchange/org.mule.examples/scatter-gather-flow-control/[Scatter-Gather Flow Control^] from Anypoint Exchange to learn more about how to use the Scatter-Gather component. This example shows the use of the scatter-gather control flow to aggregate data in parallel and return the result in JSON. - -The example uses prepared data as input for two resources to be aggregated. The data represents information about two contacts and has the following structure: - -|=== -|Resource|`firstname`|`surname`|`phone`|`email` -|contacts-1.csv -|John -|Doe -|096548763 -|+john.doe@texasComp.com+ - -|contacts-2.csv -|Jane -|Doe -|091558780 -|+jane.doe@texasComp.com+ -|=== - -The example uses DataWeave to aggregate the data. The contact information is aggregated to a JSON structure that represents data from both resources. - -To download and open this example project through Anypoint Studio, click the Exchange icon in the upper-left corner. Then, in the window that opens, log into Anypoint Exchange and search on the name of the project. - == Repeatable Stream Input The Scatter-Gather component can process only repeatable streams. Scatter-Gather cannot process a non-repeatable stream, which the scope can receive as input from an upstream component that is configured with a non-repeatable streaming strategy. From 606788450ff05b30f0d5034e3ab2274e03272521 Mon Sep 17 00:00:00 2001 From: dukesphere Date: Mon, 31 Oct 2022 22:24:19 -0700 Subject: [PATCH 13/16] W-11939773-scatterGatherEdits-duke ex --- modules/ROOT/pages/scatter-gather-concept.adoc | 17 ----------------- 1 file changed, 17 deletions(-) diff --git a/modules/ROOT/pages/scatter-gather-concept.adoc b/modules/ROOT/pages/scatter-gather-concept.adoc index 0f4476e82b..9c80f360e0 100644 --- a/modules/ROOT/pages/scatter-gather-concept.adoc +++ b/modules/ROOT/pages/scatter-gather-concept.adoc @@ -92,23 +92,6 @@ INFO ...LoggerMessageProcessor: "payload": "hello scatter gather second route", "outboundPropertyNames": [ - ], - "attributes": null - }, - "2": { - "inboundAttachmentNames": [ - - ], - "exceptionPayload": null, - "inboundPropertyNames": [ - - ], - "outboundAttachmentNames": [ - - ], - "payload": "hello scatter gather third route", - "outboundPropertyNames": [ - ], "attributes": null } From db14177c05fac7010e0596128aeddada40df843e Mon Sep 17 00:00:00 2001 From: dukesphere Date: Mon, 31 Oct 2022 22:27:07 -0700 Subject: [PATCH 14/16] W-11939773-scatterGatherEdits-duke ex --- modules/ROOT/pages/scatter-gather-concept.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/ROOT/pages/scatter-gather-concept.adoc b/modules/ROOT/pages/scatter-gather-concept.adoc index 9c80f360e0..949baf274c 100644 --- a/modules/ROOT/pages/scatter-gather-concept.adoc +++ b/modules/ROOT/pages/scatter-gather-concept.adoc @@ -55,7 +55,7 @@ To understand how scatter-gather processing of a payload occurs, assume that the * The first `` within Scatter-Gather adds the string `" first route"` to the input `payload`. * The second `` within Scatter-Gather adds the string `" second route"` to the same input `payload`. -The first Logger after the component prints the output `payload` of Scatter-Gather, which is an object containing key-value pairs for each route. The key provides the index of the route (0 or 1), and the value provides a set of key-value pairs for the payload, attributes, and other properties of the event, _excluding_ Mule variables: +The first Logger after the component prints the output `payload` of Scatter-Gather, which is an object containing key-value pairs for each route. The key provides the index of the route (`0` or `1`), and the value provides a set of key-value pairs for the payload, attributes, and other properties of the event, _excluding_ Mule variables: [source, JSON] ---- @@ -98,7 +98,7 @@ INFO ...LoggerMessageProcessor: } ---- -To access and manipulate such a payload, you can use a DataWeave function such as `mapObject`. The second Logger in the example uses `valuesOf(payload mapObject { ($$) : $}).payload` to print the payloads of each key in the output object: +To access and manipulate such a payload, you can use a DataWeave function such as xref:dataweave::dw-core-functions-mapobject.adoc[mapObject]. The second Logger in the example uses `valuesOf(payload mapObject { ($$) : $}).payload` to print the payloads of each key in the output object: [source, JSON] ---- From fd17db94322f0a8eb15acb188e0d55889db60390 Mon Sep 17 00:00:00 2001 From: dukesphere Date: Mon, 31 Oct 2022 22:44:08 -0700 Subject: [PATCH 15/16] W-11939773-scatterGatherEdits-duke ex --- modules/ROOT/pages/scatter-gather-concept.adoc | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/modules/ROOT/pages/scatter-gather-concept.adoc b/modules/ROOT/pages/scatter-gather-concept.adoc index 949baf274c..e059db03da 100644 --- a/modules/ROOT/pages/scatter-gather-concept.adoc +++ b/modules/ROOT/pages/scatter-gather-concept.adoc @@ -214,11 +214,9 @@ If you configure a timeout for a Scatter-Gather component and a route does not c == Repeatable Stream Input -The Scatter-Gather component can process only repeatable streams. Scatter-Gather cannot process a non-repeatable stream, which the scope can receive as input from an upstream component that is configured with a non-repeatable streaming strategy. +Repeatable streams are a Mule feature that enables a component to re-read a stream. Non-repeatable streams can be read only once before they are lost. By default, Mule components return a repeatable stream unless their streaming strategy is configured to be non-repeatable. -Repeatable streams are a Mule feature that enables a component to re-read a stream. Non-repeatable streams can be read only once before they are lost. - -By default, Mule components return a repeatable stream unless their streaming strategy is configured to be non-repeatable. +The Scatter-Gather component can process only repeatable streams and _cannot_ process a non-repeatable stream. If Scatter-Gather receives a non-repeatable stream as input from an upstream component that is configured with such a strategy, Scatter-Gather will not be able to process it. == Scatter-Gather XML Reference From 5595d6af24a59434bfd1e2dd01e35a3b774a57f7 Mon Sep 17 00:00:00 2001 From: dukesphere Date: Wed, 2 Nov 2022 09:06:58 -0700 Subject: [PATCH 16/16] W-11939773-scatterGatherEdits-duke log error --- modules/ROOT/pages/scatter-gather-concept.adoc | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/modules/ROOT/pages/scatter-gather-concept.adoc b/modules/ROOT/pages/scatter-gather-concept.adoc index e059db03da..c01ad4f9a1 100644 --- a/modules/ROOT/pages/scatter-gather-concept.adoc +++ b/modules/ROOT/pages/scatter-gather-concept.adoc @@ -167,15 +167,18 @@ Example of handling these errors: - + - + - + - + + + + @@ -206,6 +209,7 @@ org.mule.runtime.core.internal.message.DefaultMessageBuilder$MessageImplementati attributes= attributesMediaType=*/* } +WARN ...LoggerMessageProcessor: COMPOSITE_ROUTING ---- == Handle Timeout Errors in a Scatter-Gather