Parallel processing significantly improves integration performance by enabling concurrent execution of multiple flows. Parallel processing with Scatter-Gather in MuleSoft enhances performance by executing flows concurrently. However, managing and preserving errors that occur in these parallel branches requires careful consideration. This post explores strategies for effectively handling and tracking errors in Scatter-Gather.
Understanding Scatter-Gather error handling
Scatter-Gather distributes a message across multiple routes. Errors in individual routes don’t immediately halt the overall process. By default, Scatter-Gather collects results from all routes, including error responses. After completing the flow, a generic error message is displayed with error code as MULE:COMPOSITE_ROUTING.
Scatter-Gather challenges
- Error handling: Unhandled errors in one route can halt the entire processing flow
- Variable scope: Variable modifications within the scatter-gather scope are not retained afterward
- Error object clarity: Insufficient detail in the error object complicates exception troubleshooting
What we'll cover in this article: We’ll outline a method for capturing and presenting detailed error information and descriptions to API consumers.
Error handling and debugging in Scatter-Gather
We recommend having a dedicated “on error handler” within your error handling process to log or respond with the underlying details of composite routing errors. By default, composite error handling does not log these underlying details.
The variables below will not be available in a debug window, but can be used to retrieve the required information. When handling errors within a Mule application, particularly when utilizing a Scatter-Gather component, understanding how to access and interpret error messages is crucial for effective debugging and logging.
The expression `error.errorMessage.payload.failures` serves as a powerful tool for obtaining a comprehensive overview of all failures that have occurred. This expression provides a complete matrix of failures, allowing developers to see detailed information about each individual sub-flow or route that encountered an issue.
For more targeted analysis, specifically to identify which routes within a Scatter-Gather component have failed, the expression `error.errorMessage.payload.failures pluck $$` is highly effective. This DataWeave expression extracts a concise list of the failed route numbers. It’s important to note that these route numbers are zero-indexed, meaning the first route is “0,” the second is “1,” and so on.
A significant consideration when implementing error logging, especially in conjunction with Scatter-Gather, is the presence of an “until successful” scope. If an “until successful” scope is embedded within a Scatter-Gather implementation, the structure and content of the error logging expressions can change significantly. This is because “until successful” attempts to reprocess messages until they succeed, which can alter the state and availability of error information at the time of logging.
Here are some examples of DWL implementation to capture route-based errors:
Error map (without until successful in routes)
%dw 2.0
output application/java
var routesMeta = ["Route1", "Route2", "Route3", "Route4", "Route5"]
var routes = error.errorMessage.payload.failures pluck $$
---
{
failingComponent: error.failingComponent,
failingRoutes: routes map (item, index) -> {
errorRoute: routesMeta[item as Number],
errorDescription: error.errorMessage.payload.failures[index].exception.message
}
}Error map (using until successful in routes)
%dw 2.0
output application/java
var routesMeta = ["Route1", "Route2", "Route3", "Route4", "Route5"]
var routes = error.errorMessage.payload.failures pluck $$
---
{
failingComponent: error.failingComponent,
failingRoutes: routes map (item, index) -> {
errorRoute: routesMeta[item as Number],
errorDescription: error.errorMessage.payload.failures[index].suppressedErrors.description
}
}Sample error handling to log or respond these errors can be like the following:
<on-error-propagate enableNotifications="true" logException="true" doc:name="On Error Propagate"
doc:id="0497cbf9-092f-4391-8b65-b64310b2cfff" type="MULE:COMPOSITE_ROUTING">
<ee:transform doc:name="Extract error" doc:id="364f226b-d20c-43c6-9d17-77b342195df0">
<ee:message>
</ee:message>
<ee:variables>
<ee:set-variable variableName="compositeError">
<![CDATA[%dw 2.0
output application/java
var routesMeta = ["Route1", "Route2", "Route3", "Route4", "Route5"]
var routes = error.errorMessage.payload.failures pluck $$
---
{
failingComponent: error.failingComponent,
failingRoutes: routes map (item, index) -> {
errorRoute: routesMeta[item as Number],
errorDescription: error.errorMessage.payload.failures[index].suppressedErrors.description
}
}]]>
</ee:set-variable>
</ee:variables>
</ee:transform>
</on-error-propagate>Conceptual implementation example
Consider a practical application where the Scatter-Gather component can significantly streamline data processing. An incoming request needs to retrieve information from multiple disparate systems, e.g. multiple objects from a Data Cloud system.
In this use case, the Scatter-Gather component acts as an orchestrator. Upon receiving the initial request, it can concurrently dispatch messages to several distinct routes, each designed to interact with a specific external system. For instance:
- Route 1: Source Accounts
- Route 2: Account Details
- Route 3: Account Hierarchy
- Route 4: Account Address
- Route 5: Business Relationship
To manage these concurrent operations efficiently, a variable would be created, containing a list of these route names. This allows for dynamic configuration and easier management of the various data sources involved. Once each route completes its task, the Scatter-Gather component then aggregates all the individual responses into a single, consolidated message.
This aggregated message provides a comprehensive view of the customer’s request, drawing data from all relevant systems in parallel, thereby significantly reducing the overall processing time compared to sequential calls.
In the scenario above, a sample DWL to prepare an error message might be:
%dw 2.0
output application/java
var routesMeta = ["Source Accounts", "Account Details", "Account Hierarchy", "Account Address", "Business Relationship"]
var routes = error.errorMessage.payload.failures pluck $$
---
{
errorType: error.errorMessage.payload.failures.suppressedErrors.errorType default { "namespace":"MULE","identifier": "COMPOSITE_ROUTING"},
description: error.errorMessage.payload.failures.suppressedErrors.description default "An Exception Occured",
failingComponent: error.failingComponent,
failingRoutes: routes map (item, index) -> {
errorRoute: routesMeta[item as Number],
errorDescription: error.errorMessage.payload.failures[index].suppressedErrors.description
}
}Sample response payload which shows with list of failed routes:
{
"message": "An unexpected error occurred. Please try again later.",
"detail": "Testing route error. Please contact support with the transaction ID: 0e6e37f6-31b1-461d-86ef-0f6d5e61b6a3",
"failingComponent": "ent-api-business-entity-main/processors/4 @ ent-api-business-entity:ent-api-business-entity.xml:29",
"failingRoutes": [
{
"errorRoute": "account details",
"errorDescription": "SDC:BAD_REQUEST Query execution failed at Data Cloud"
},
{
"errorRoute": "account hierarchy",
"errorDescription": "Timeout exceeded"
}
]
}
]
}
}Enhanced error capture in applications
Here’s an alternative approach for error handling, which can be integrated with any existing error handling method. This code snippet is designed to capture the first error encountered within the application. A default error message is included to ensure a comprehensive and reliable response.
Error snippet:
%dw 2.0
output application/java
var firstFailure = error.errorMessage.payload.failures[0] default {}
var firstChildError = firstFailure.childErrors[0] default {}
var firstSuppressedError = firstChildError.suppressedErrors[0] default {}
---
{
"errorType": firstSuppressedError.errorType
default firstChildError.errorType
default firstFailure.errorType
default { "namespace":"MULE","identifier": "COMPOSITE_ROUTING"} ,
"description": firstChildError.description
default firstSuppressedError.description
default firstFailure.description
default "An Exception Occurred",
failingComponent: error.failingComponent
}Preserving errors in MuleSoft Scatter-Gather
Effectively preserving errors in MuleSoft Scatter-Gather is essential for building robust and reliable integrations. By implementing clear error handling strategies and comprehensive logging, you can efficiently manage parallel processing challenges and ensure the integrity of your data flows.
