Streaming in Mule

May 28 2009

4 comments 0

Streaming enables efficient processing of large data objects such as files, documents, and records by streaming the data through Mule rather than reading the whole thing into memory. Streaming provides the following advantages:

  • Allows services to consume very large messages in an efficient way
  • Message payloads are not read into memory
  • Simple routing rules based on message metadata are still possible
  • You can combine streaming and non-streaming endpoints

Streaming Transports

The following Mule transports support streaming:

Streaming Transformers and Filters

Many Mule transformers and filters can read input streams, process the contents, and send them on. However, most of these do not process the stream in real time; instead, they read the stream, load it into memory, process it, and then send it on. Therefore, transformers and filters can become a bottleneck in your application if you regularly stream large files.

The following transformers and filters do support true streaming and process the data as streams without loading them into memory first:

We'd love to hear your opinion on this post

4 Responses to “Streaming in Mule”

  1. – XSLT Transformer

    Chained xsl transformers are not streamed at all in Mule.

  2. Hi Jackie
    What if the stream is dispatched to a tapper ? I mean for example, a CXF inbound endpoint and a wire tap on JMS. For my experiences if the stream is consumed it won’t be more available..
    In case of multicast outbound ?

  3. Hi Alessio,
    Good question :). In the scenario you describe the message will be read into memory and a copy will be sent to JMS and one to the Web Service. Obviously, this defeats the purpose of streaming, but for SOAP messages it is less likely that the SOAP envelope will be huge. Note that JMS does not support streaming in that you can’t stream a message directly to the JMS server.

    Rule of thumb, if you are processing very large messages in Mule, you need to use point to point routing so that the stream flow can be preserved. You can still to conditional routing but make sure that the current message is routed to a single location.

  4. […] these existing security features while leveraging the benefits associated with Mule flows such as support for streaming and the Mule Expression Language. With that in mind, let’s take a tour of the new features that […]