Reading Time: 27 minutes

This post was written by one of the stars in our developer community, Manik Magar 

Mule 4 beta is already out. One of the major change in Mule 4 is making DataWeave a default expression language over Mule 3’s default Mule Expression Language (MEL). The XML namespace of DataWeave is moved from dw to ee (core), and the DataWeave version has changed from 1.0 to 2.0 in Mule 4.

latest report
Learn why we are the Leaders in API management and iPaaS
Read reports

Apart from syntax changes, there are many new features in DataWeave 2.0. In a series of two blog posts, I will focus on the syntax changes in DataWeave 2.0 and compare them to 1.0. 

In this blog post, I will specifically discuss header changes in DataWeave (section 1) as well as iterative function changes within the body (section 2). In my next blog post, I will dive deeper into DataWeave body changes by discussing operator changes as well as other differences.

Changes covered in this post may not be the list of all syntax changes and there may be more changes.
 
Let’s begin with a quote about change: “Your success in life isn’t based on your ability to simply change. It is based on your ability to change faster than your competition, customers, and business.”

Now let’s get started, a DataWeave script file can be divided into two sections:

  1. Header that defines directives
  2. Body that defines the output

1. DataWeave Header Changes

The DataWeave version has changed from 1.0 to 2.0. Along with that, the list below summarizes changes to the header sections of DataWeave:

  1. dw version changed from %dw 1.0 to %dw 2.0
  2. All directives in DataWeave 1.0 must start with %. This is no longer needed in 2.0 (except for %dw declaration).
  3. Custom functions in DataWeave 1.0 are defined with function directive. In DataWeave 2.0, it has been changed to fun. Also, the function body and function signature must separate by = instead of a space.
  In both dataweave versions, you can define custom functions with either var or function(/fun) directives.

In my honest opinion, all these changes may be trivial but they are good. The removal of %, saves you some characters and makes the header section look more like regular code instead of directives.

The two code listings below show the same script written in both versions:

Listing:1.A – DataWeave 1.0 Headers
%dw 1.0
%output application/json
%var user = {"firstName": "Manik", "lastName" : "Magar"}
%var foo = "bar"
%input payload application/json
%function getName(data) (data.firstName ++ ' ' ++ data.lastName)
%var toFullname = (data) -> (data.firstName ++ ' ' ++ data.lastName)
---
{
"foo" : foo,
"getName" : getName(user),
"toFullname" : toFullname(user)
}
Listing:1.B – DataWeave 2.0 Headers
%dw 2.0
output application/json
input payload application/json
var user = {"firstName": "Manik", "lastName" : "Magar"}
var foo = "bar"
fun getName(data) = (data.firstName ++ ' ' ++ data.lastName)
var toFullname = (data) -> (data.firstName ++ ' ' ++ data.lastName)
---
{
"foo" : foo,
"getName" : getName(user),
"toFullname" : toFullname(user)
}
Listing:1.C – Output of Both Scripts
{
  "foo": "bar",
  "getName": "Manik Magar",
  "toFullname": "Manik Magar"
}

2. DataWeave Body Changes

DataWeave script’s body section is where you write all your transformation logic that defines output structure. Some interesting changes have happened to the body section of the script as well. Let’s look at those changes.

  Below grouping is not an official grouping by MuleSoft but just something I came up with. Makes it easy to write and follow :).

2.1 Iterative Functions

You guessed it right! These are the functions that allow you to iterate and operate on a collection of objects or a single object. This can include functions like map,mapObject, filter, pluck, groupBy, etc. Some interesting changes have been made in DataWeave 2.0 for these functions.

2.1.1 Automatic Coercion from Object to Array

DataWeave 1.0 allowed automatic coercion of object to array. What that means, is in DataWeave 1.0, the map operator which allows you to iterate over array or collection of objects, was also working if the input was an object.

Consider the DataWeave 1.0 script below, where we expected to receive a list of users to map. For some reason, at runtime we received a single user object instead of the collection of users. Why? Because bugs are everywhere!.

Our script should have either failed or generated single user object, but it generates two user objects with some really incorrect data.
 
Listing:2.1.1.A – DataWeave 1.0 Object to Array Coercion
%dw 1.0
%output application/json
%var users = {"firstName": "Manik", "lastName": "Magar"}
---
users map {
"firstName": $.firstName,
"lastName": $.lastName
}

Listing:2.1.1.B – The Output

[
  {
    "firstName": "Manik",
    "lastName": null
  },
  {
    "firstName": null,
    "lastName": "Magar"
  }
]
Are you wondering why?

Because DataWeave 1.0 treated every property (key-value pair) of an object as a single object and then our input object to be a collection of those objects, like the array in the listing below. Now, if you provide this as an input to map, you are going to get two objects with some null fields.

But was that the expected output from original script? Off course, Not.

Listing:2.1.1.C – Object Coerced to an Array
[
  {
    "firstName": "Manik"
  },
  {
    "lastName": "Magar"
  }
]

DataWeave 2.0 has generously corrected this behavior and removed the automatic coercion of objects to array. With, DataWeave 2.0, a similar script will generate an error (in preview mode, as well as at runtime) 

mule 4 dataweave
 
Did you notice the error details shown in studio 7? They are so detailed and clear! Yet Another Useful Change!

In my opinion, this change can definitely safeguard users from having unexpected output and data corruption in the downstream systems. 

2.1.2 Overloaded Functions

In DataWeave 2.0, the functions filter and groupBy are overloaded to operate on objects. This could be useful to create sub-objects from a large object.

Consider an object {"firstName": "Manik", "lastName": "Magar", "Address Line1" : "Address line 1"}and we want to create another object with all the properties where value contains ‘Ma’.

With DataWeave 1.0, we will need to use mapObject and then map all key-value pairs conditionally. Something like the listing below:

Listing:2.1.2.A – DataWeave 1.0 Creating Sub-objects
%dw 1.0
%output application/json
%var user = {"firstName": "Manik", "lastName": "Magar", "Address Line1" : "Address line 1"}
---
name : user mapObject {
('$' : '
Listing:2.1.1.B – The Output
{
  "name": {
    "firstName": "Manik",
    "lastName": "Magar"
  }
}

In DataWeave 2.0, we can leverage the overloaded function of filter which allows us to operate on objects and achieve the same result but with shorter code.

Listing:2.1.1.C – DataWeave 2.0 Creating Sub-objects with a Filter Function
%dw 2.0
output application/json
var user = {"firstName": "Manik", "lastName": "Magar", "Address Line1" : "Address line 1"}
---
name : user filter ($ contains "Ma")

Above one-line and simple code generates exact same output as in Listing:2.1.1.B. To understand the difference, if you write that filter function in DataWeave 1.0, the output will not be an object but an array of values satisfying the given condition.

Listing:2.1.1.D - DataWeave 1.0 Filter Object Output
{
  "name": [
    "Manik",
    "Magar"
  ]
}

Similarly, groupByfunction has been overloaded to allow invoking three-argument lambda on object:

Listing:2.1.1.E - DataWeave 2.0 Group by an Object with an Overloaded Function
%dw 2.0
output application/json
var user = {"firstName": "Manik", "lastName": "Magar", "Address Line1" : "Address line 1"}
---
user groupBy ((value, key, index) -> if (key contains 'Name') "Names" else "Address")
  Listing 2.1.1.E also gives you a sneak peak at if …​ else …​ that replaces when …​ otherwise …​ conditional syntax of DataWeave 1.0. We will see more about it in a later part.

Running this script against our user object, will partition (group) the object as below -

Listing:2.1.1.F - The Output
{
  "other": {
    "Address Line1": "Address line 1"
  },
  "Names": {
    "firstName": "Manik",
    "lastName": "Magar"
  }
}
  In DataWeave 1.0, the lambda signature is (value, index) so it is not possible to write a similar group by on an object.

If you run groupBy on an object in DataWeave 1.0, it does not output keys which could be another problem.

Listing:2.1.1.G - DataWeave 1.0 Group by an Object
%dw 1.0

%output application/json
%var user = {"firstName": "Manik", "lastName": "Magar", "Address Line1" : "Address line 1"}
---
user groupBy ((value, index) -> "Name" when (value contains 'Ma') otherwise "Address")
Listing:2.1.1.H - The Output
{
  "Address": [
    "Address line 1"
  ],
  "Name": [
    "Manik",
    "Magar"
  ]
}

To achieve an output like Listing:2.1.1.F in DataWeave 1.0, we will need to take an approach similar to that shown in Listing:2.1.2.A.

2.1.3 Adding a Third Parameter

DataWeave 2.0 has added index as a 3rd parameter to mapObject, pluck, filter, and groupBy. Some of these functions also have index in DataWeave 1.0, but as a second parameter.

Consider the two code listings below:

Listing:2.1.3.A - DataWeave 1.0 New Parameter Addition
%dw 1.0
%output application/json
%var user = {"firstName": "Manik", "lastName": "Magar", "addressLine1" : "Address line 1"}
%var userList = [{"firstName": "Manik", "lastName": "Magar", "addressLine1" : "Address line 1"},
{"firstName": "Om", "lastName": "Magar", "addressLine1" : "Address line 2"}
]
---
{
mapObject: user mapObject (value, key) -> {    
'$key': value
},

pluck: user pluck (value, key) -> (key ++ '-' ++ value),   

filter: user filter ((value, index) -> (index < 1)),   
filter2: userList filter (value, index) -> (value.firstName contains 'Ma'),    

groupBy1 : user groupBy ((value, index) -> (index < 1)),     
groupBy2 : userList groupBy ((value, index) -> value.lastName)     

}
Listing:2.1.3.B - DataWeave 2.0 New Parameter Addition
%dw 2.0
output application/json
var user = {"firstName": "Manik", "lastName": "Magar", "addressLine1" : "Address line 1"}
var userList = [{"firstName": "Manik", "lastName": "Magar", "addressLine1" : "Address line 1"},
{"firstName": "Om", "lastName": "Magar", "addressLine1" : "Address line 2"}
]
---
{
mapObject: user mapObject (value, key, index) -> {   
'$key-$index': value
},

pluck: user pluck (value, key, index) -> (key ++ '-' ++ index ++ '-' ++ value),    

filter: user filter ((value, key, index) -> (index < 1)),    
filter2: userList filter ((value, index) -> (index < 1)),    

groupBy1 : user groupBy ((value, key, index) -> (index < 1)),    
groupBy2 : userList groupBy ((value, index) -> value.lastName)   

}

Listing 2.1.3 - A vs B

  • The third parameter index has been added to mapObject function. This could be useful if you want to iterate on object properties based on property indices. 
  • There is an addition of a third index parameter to pluck, similar to mapObject. 
  • The filter function had index parameter in DataWeave 1.0 too, but when an object is passed to filter function, the index moves to the third place and a new second param key is introduced to the lambda. This is a great addition to operate over objects. 
  • There is NO change to filter lambda when input is a collection. 
  • Similar to filter on an object, groupBy for an object also gets key as new second param and index moved to third. 
  • There is NO change to gruopBy when input is a collection.

2.1.4 Allow NULL Input

Things happen, payloads becomes null sometime and when that happens, script starts failing. In DataWeave 1.0, functions will error out due to null input.

In DataWeave 2.0, map, mapObject and filter functions have been modified to accept NULL input and when that happens, they just return NULL.

Consider the DataWeave 1.0 and 2.0 scripts below: 

Listing:2.1.4.A - DataWeave 1.0 Null Input
{!{code class="hljs javascript"}!}czoxODU6XCIlZHcgMS4wJWR3IDEuMCUKb3V0cHV0IGFwcGxpY2F0aW9uL2pzb24KLS0tCnsgCiB1c2VyOiBudWxsIGZpbHRlciAoJC57WyYqJl19Zmlyc3ROYW1lID09IFwnTWFuaWtcJyksCiAKIHVzZXI6IG51bGwgbWFwT2JqZWN0IHsgCiAgIFwnJCRcJzogJCAKIH0sCiB1c2VycyA6IG57WyYqJl19dWxsIG1hcCB7IAogICAgICAgbmFtZTogJC5maXJzdE5hbWUgCn0KfVwiO3tbJiomXX0={!{/code}!}

All of the above mappings in DataWeave 1.0 will fail to compile due to NULL input.

Listing:2.1.4.B - DataWeave 1.0 Null Input
%dw 2.0
output application/json
---
{
user: null filter ($.firstName == 'Manik'),

user: null mapObject {
'$': $
},

users : null map ((value, index) -> value.firstName)
}

All these mappings in DataWeave 2.0 will not fail but will generate the output below.

  Whether this is acceptable output or not, depends on your use case. You should consider the null input case and handle it appropriately.
 
Listing:2.1.4.B - Output of DataWeave 2.0
{!{code class="language-json hljs" data-lang="json"}!}czo0ODpcInsKIFwidXNlclwiOiBudWxsLAogXCJ1c2VyXCI6IG51bGwsCiBcInVzZXJzXCI6IG51bGwKfVwiO3tbJiomXX0={!{/code}!}
  NULL for map is allowed when lambda is used and returns a single value that can be made null i.e. (T, Number) → Boolean format. For example, consider below mapping that tries to output a key-value pair, but will error out in DataWeave 2.0 too. 
users : null map ((value, index) -> (name: value.firstName))

And that summarizes DataWeave changes in headers and in the body (regarding iterative functions)! Check out my upcoming post, where I will further discuss DataWeave 2.0 body changes by focusing on operator changes as well as other differences.

This article first appeared on JavaStreets.

 

) when ($ contains 'Ma') }

Listing:2.1.1.B - The Output
 

In DataWeave 2.0, we can leverage the overloaded function of filter which allows us to operate on objects and achieve the same result but with shorter code.

Listing:2.1.1.C - DataWeave 2.0 Creating Sub-objects with a Filter Function
 

Above one-line and simple code generates exact same output as in Listing:2.1.1.B. To understand the difference, if you write that filter function in DataWeave 1.0, the output will not be an object but an array of values satisfying the given condition.

Listing:2.1.1.D - DataWeave 1.0 Filter Object Output
 

Similarly, groupByfunction has been overloaded to allow invoking three-argument lambda on object:

Listing:2.1.1.E - DataWeave 2.0 Group by an Object with an Overloaded Function
 
  Listing 2.1.1.E also gives you a sneak peak at if …​ else …​ that replaces when …​ otherwise …​ conditional syntax of DataWeave 1.0. We will see more about it in a later part.

Running this script against our user object, will partition (group) the object as below -

Listing:2.1.1.F - The Output
 
  In DataWeave 1.0, the lambda signature is (value, index) so it is not possible to write a similar group by on an object.

If you run groupBy on an object in DataWeave 1.0, it does not output keys which could be another problem.

Listing:2.1.1.G - DataWeave 1.0 Group by an Object
 
Listing:2.1.1.H - The Output
 

To achieve an output like Listing:2.1.1.F in DataWeave 1.0, we will need to take an approach similar to that shown in Listing:2.1.2.A.

2.1.3 Adding a Third Parameter

DataWeave 2.0 has added index as a 3rd parameter to mapObject, pluck, filter, and groupBy. Some of these functions also have index in DataWeave 1.0, but as a second parameter.

Consider the two code listings below:

Listing:2.1.3.A - DataWeave 1.0 New Parameter Addition
 
Listing:2.1.3.B - DataWeave 2.0 New Parameter Addition
 

Listing 2.1.3 - A vs B

  • The third parameter index has been added to mapObject function. This could be useful if you want to iterate on object properties based on property indices. 
  • There is an addition of a third index parameter to pluck, similar to mapObject. 
  • The filter function had index parameter in DataWeave 1.0 too, but when an object is passed to filter function, the index moves to the third place and a new second param key is introduced to the lambda. This is a great addition to operate over objects. 
  • There is NO change to filter lambda when input is a collection. 
  • Similar to filter on an object, groupBy for an object also gets key as new second param and index moved to third. 
  • There is NO change to gruopBy when input is a collection.

2.1.4 Allow NULL Input

Things happen, payloads becomes null sometime and when that happens, script starts failing. In DataWeave 1.0, functions will error out due to null input.

In DataWeave 2.0, map, mapObject and filter functions have been modified to accept NULL input and when that happens, they just return NULL.

Consider the DataWeave 1.0 and 2.0 scripts below: 

Listing:2.1.4.A - DataWeave 1.0 Null Input
 

All of the above mappings in DataWeave 1.0 will fail to compile due to NULL input.

Listing:2.1.4.B - DataWeave 1.0 Null Input
 

All these mappings in DataWeave 2.0 will not fail but will generate the output below.

  Whether this is acceptable output or not, depends on your use case. You should consider the null input case and handle it appropriately.
 
Listing:2.1.4.B - Output of DataWeave 2.0
 
  NULL for map is allowed when lambda is used and returns a single value that can be made null i.e. (T, Number) → Boolean format. For example, consider below mapping that tries to output a key-value pair, but will error out in DataWeave 2.0 too. 
 

And that summarizes DataWeave changes in headers and in the body (regarding iterative functions)! Check out my upcoming post, where I will further discuss DataWeave 2.0 body changes by focusing on operator changes as well as other differences.

This article first appeared on JavaStreets.