Getting Started with DataWeave: Part 4

September 29 2015

20 comments 0
Getting started with dataweave part 4

In this final part of our introduction to DataWeave, we now present you with some example transformations. Just in case you missed the previous posts I included parts 1 to 3 below:

We will use everything we have learned till now to realize the transformations. Wherever we introduce new concepts we will highlight them in a subsequent set of notes. So, without further delay, let’s jump in!

Scenario 1

Google’s Address API responds with a JSON object which contains an array of addresses followed by their distances from our current location. We want to transform this into an array of the top 5 closest addresses.

Input

Screen Shot 2015-09-23 at 7.41.36 PM

Output

Screen Shot 2015-09-23 at 7.41.45 PM

Transformation

Screen Shot 2015-09-23 at 8.31.04 PM

Notes

  • The flatten operator transforms an array of arrays to a single array containing all of the elements in each of the nested arrays.
  • $$ is an alias for the the index of the current iteration through a.
  • d[$$] selects the element in whose index is $$. Dynamic key selection will also work on objects. Any expression which resolves to the name of the key can be used within square brackets.

Scenario 2

Given an Xml document with a sequence of line items, transform it to a JSON document with the field names slightly modified.

Input

Screen Shot 2015-09-23 at 7.26.36 PM

Output

Screen Shot 2015-09-23 at 7.37.26 PM

Transformation

Screen Shot 2015-09-24 at 8.07.48 AM

Notes

  • ‘$$_1’ allows us to dynamically build the value for the keys in our output by appending ‘_1’ to the key name present in the input.
  • mapObject is an operator that iterates through the key:value pairs on the input Value object and rather than produce an array, it simply builds another object defined by its right-hand operand.

Scenario 3

Given 2 responses to SOAP-based WebServices as input transform them into JSON object. The first Xml GetMedicationsResponse contains the medications for a patient. The second Xml GetMedicationsOrderResponse contains orders for the same medications. We need to iterate through each Medication in the first Xml and only produce output if the InternalId for the Medication is in the list of Orders and they Route code for the Order is oral (15) and the RefillsRemaining on the Order is greater than 1.

Input

Screen Shot 2015-09-23 at 7.42.20 PMScreen Shot 2015-09-23 at 7.42.30 PM

Output

Screen Shot 2015-09-23 at 7.42.40 PM

Transformation

Featured Image DataWeave 4

Notes

  • The %var global variable declaration allows us to use those variables anywhere in the , including other global variable value assignments and function bodies.
  • The [?(<filter-criteria>)] expression allows us to filter on a key selector. There is a subtle difference between this mode of filtering and that of the filter operator. When no elements on the array match the criteria in this construct, then null is returned. Filter on the contrary will return an empty array.
  • The contains operator tests for the presence of a value in a container. The right-hand operand is the value. The left-hand operand is the container. This can be an array, a string or an object. When objects are tested, the container is actually the array of values in its sequence of key:value pairs. Here, we simply test arrays of strings for the presence of a string.

Scenario 4

Given a plain text sequence of identifiers, separated by the character ‘+’, and each of which may begin with a code containing 2 to 4 upper-case letters, output a JSON object whose keys are those same codes and whose values are arrays of the remaining part of each identifier which begins with the code. If any identifiers are found that don’t contain codes as specified, then these should be added to the array keyed by ‘invalid’ in the output.

Input

Screen Shot 2015-09-23 at 7.42.01 PM

Output

Screen Shot 2015-09-23 at 7.42.11 PM

Transformation

Screen Shot 2015-09-23 at 8.47.45 PM

Notes

  • The splitBy operator creates an array of the values resulting from those sequences of characters between each occurance of the splitting criteria specified as its right hand operand.
  • The match operator returns an array of matches that contains the entire matching expression, followed by all of the capture groups that match the provided regular expression. In this case ($ match /([A-Z]{2,4})\d*/)[1] will return the code which will be in the first and only capture  group for those identifiers it matches.
  • The default operator returns its right hand operand value when its left-hand operand returns null.
  • The pluck operator iterates through the key:value pairs on an object and builds an array of its values ($) or its keys ($$).

Scenario 5

Given an Xml document containing Stock quotes, transform it into a plain text sequence of characters with the <Values/> elements padded to 4 digits before and after the decimal, separating each Quote with a ‘~’ and terminating the SymbolId with a ‘*’.

Input

Screen Shot 2015-09-24 at 4.35.43 PM

Output

Screen Shot 2015-09-24 at 4.35.54 PM

Transformation

Screen Shot 2015-09-29 at 1.35.03 PM

Notes

  • The reduce operator iterates through an array and executes the lamda function you define as its right-hand operand. In this case we append each element quote to the accumulator qStr.

Scenario 6

Take the output from Scenario 6, the plain text string of codes and values and transform it back to the original input Xml from Scenario 5.

Input

Screen Shot 2015-09-24 at 4.35.54 PM

Output

Screen Shot 2015-09-24 at 4.35.43 PM

Transformation

Screen Shot 2015-09-25 at 2.44.57 PM

Notes

  • Here we necessarily cast each of the values to a number with two decimal places using the as operator. Thus we remove the padding zeros leading and trailing each value.

Further Reference

That’s it! We refer you to our documentation for an deep reference of the complete language as well as a tutorial to help you get started and a webinar with more a live demonstration and finally, some more examples.


 


We'd love to hear your opinion on this post

20 Responses to “Getting Started with DataWeave: Part 4”

  1. Hi Nial Darbey, Thanks for the great article – can you please elaborate on scenario 3 – how can the data weave transformer consume two XML payload’s ?

    Agree(0)Disagree(0)Comment
    • Hi Ghulam,
      The DataWeave engine receives the entire Message which includes not just the payload but all of the inbound properties, outbound properties and the variables which have been explicitly set by the developer prior to the transformer. DataWeave allows you to access these through the reserved variable names like payload and flowVars. Mule allows you to enrich the Message as it passes through each Message Processor in the flow, so you can imagine receiving the Message from a call to your http listener, or perhaps receiving it from a call to the Database or driven by the arrival of a message in a Message Broker queue and then, with no limitation, you can call out to more external services / apis along the flow and store the responses from those service calls as flow variables. Thus, there is no limit to the amount of “payloads” that you can process. At any one time there is only one payload on the Mule Message, but there is no limit to the amount of variables attached to the same Message.
      Does that make sense?

      Agree(2)Disagree(0)Comment
  2. Very nice explanation..easy one..

    Agree(0)Disagree(0)Comment
  3. One thing I’ve failed to see addressed in all the docs and DataWeave examples provided by MuleSoft is code reuse. Take Scenario 3 for example, `meds` represents a core entity that is likely used application-wide in multiple transformations.

    How would one render the `meds` object and use the `ok` function across
    multiple transformations? It seems the only option is to move these into a MEL function or a Java method wrapped by a MEL function.

    For the `ok` function this may be fine, but for `meds` this presents issues since the incoming type in the weave is dependent on various factors in the flow. Seems like the only solution there is to transform incoming payloads into a canonical representation and operate on that in all the weaves.

    Agree(0)Disagree(0)Comment
  4. Nice explanation on Dataweave.. good thing, now it can handle any type of request.. flow as part of live streaming to do the operations on the data..

    Agree(0)Disagree(0)Comment
  5. Nial thanks for your explanations.
    I need to flatten some data, but I wasn’t able to do it following the example of scenario 1.
    I have the following data
    {
    “containers”:
    [
    {“containerId”: “1”, “description”: “aluminum can”, “containerSizes”:
    [
    {“containerSizeId”: “1”, “size”: “300 ml”},
    {“containerSizeId”: “2”, “size”: “500 ml”}
    ]
    },
    {“containerId”: “2”, “description”: “glass bottle”, “containerSizes”:
    [
    {“containerSizeId”: “3”, “size”: “700 ml”}
    ]
    }
    ]
    }

    and I want to convert them to something like this

    [
    {
    “containerId”: “1”,
    “description”: “aluminum can”,
    “size”: “300 ml”
    },
    {
    “containerId”: “1”,
    “description”: “aluminum can”,
    “size”: “500 ml”
    },
    {
    “containerId”: “2”,
    “description”: “glass bottle”,
    “size”: “700 ml”
    }
    ]

    I tried this:

    using (c = payload.containers,
    s = (flatten payload.containers.containerSizes)
    )
    (c map {
    containerId: $.containerId,
    description: $.description,
    size: s[$$].size
    })

    But I got this result

    [
    {
    “containerId”: “1”,
    “description”: “aluminum can”,
    “size”: “300 ml”
    },
    {
    “containerId”: “2”,
    “description”: “glass bottle”,
    “size”: “500 ml”
    }
    ]

    So I did not get the two sizes for the aluminum can container.
    Could you tell me how to do this?
    Thanks for your help.

    Agree(0)Disagree(0)Comment
    • Hi Sandra,

      Try this:

      %dw 1.0
      %output application/json
      %function expandSize(container, size)
      size.containerCharges map {
      containerId: container.containerId,
      description: container.description,
      containerSize: size.size,
      effectiveDate: $.effectiveDate,
      expiryDate: $.expiryDate default ”,
      charge: $.charge
      }
      %function expandContainer(container)
      flatten (container.containerSizes map expandSize(container, $))

      {
      containers: flatten (payload.containers map expandContainer($))
      }

      Agree(1)Disagree(0)Comment
  6. I’m trying to use the matches operator with a string (or variable) and doesn’t work.
    This isnt supported?

    Ex:
    flowVars.brands filter ($.name matches /”PEPSI”/)

    Agree(0)Disagree(0)Comment
  7. How can I access the inbound properties, query parameters using dataweave?
    Thanks

    Agree(0)Disagree(0)Comment
    • Hi Kushal,
      Just use the context variable inboundProperties
      Nial

      Agree(0)Disagree(0)Comment
  8. Thanks for this post. It really helped me understand how to use DataWeave better.

    Agree(0)Disagree(0)Comment
  9. Hi Nial,
    Could you please help me with below query? i have an xml, where in i want to extract the LoadReferenceNumber and assign to a Global variable when the LoadReferenceNumberType is “MB” in data weave.

    XML:

    MB
    8070296

    ACT
    2016

    SP
    201704

    Agree(0)Disagree(0)Comment
    • Hi Jitendra,

      Looks like your xml got swallowed up in the rendering here. Any other way you can explain what you want?
      Nial

      Agree(0)Disagree(0)Comment
  10. DataWeave component is really nice and powerful but i dint like that it actually writes mapping code in line to the flow xml. is there a way to externalize the mapping? i mean if i have a scenario where i have to map every xml tag to a new xml tag (XML transformation) and if i have too many fields to map then my flow xml will be too big.

    if there is a way to externalize this mapping to a text file or some other type of file, which can be stored under some resources folder then flow xml looks clean, nice and better organized…

    Agree(0)Disagree(0)Comment
    • Hi Raj,
      Yes you can. Check our docs here.
      Nial

      Agree(2)Disagree(0)Comment
      • Hi,
        nice info on dataweave. what if I want to read something from properties file in dataweave?

        Agree(0)Disagree(0)Comment
        • Hi Deepak,
          Just use the p(‘propertyname’) syntax.
          Nial

          Agree(0)Disagree(0)Comment
      • I need to convert csv data to Java LinkedHashMap type. But Whenever I tried to convert this data result comes out as an Arraylist. Can you kindly shed some light on this issue.?

        Thanks in Advance

        Agree(0)Disagree(0)Comment
        • Hi Sumanta,
          It makes more sense to convert to a list of maps. Each record in the csv is a map. The numerous maps are collected together in a list. Regardless of whether the record count is 1 or more, it makes sense to use a list.
          Any of the maps on the list can be accessed with the list[index] syntax.
          Nial

          Agree(0)Disagree(0)Comment