Introducing the Mule Requester Module

motif

Today I am going to introduce a recently created module: Mule Requester.

As its name may hint, its goal is to allow the request of a resource at any point in a flow. This resource can be a file, a message (from VM, JMS, AMQP, etc.), an e-mail, etc. It’s intended for resources that originally could only be requested by message sources.

Let’s try to explain it better with an example. Say we want to consume messages from a queue on demand, i.e. not consuming the message as soon as it’s put on the queue but at a later stage, when a user calls an HTTP inbound endpoint, for example. We cannot achieve this by using a JMS inbound endpoint, since it will consume the message as soon as it’s put on the queue. Thinking about a way of doing this, we could have a stopped flow and activate it on demand but this would cause the consumption of more than one message or a clumsy implementation that would pick a message and stop the flow again. Another option would be to use a component but this would have to deal with the specifics of the transport, leading to either one implementation per transport type or a big component handling all the transports.

The above mentioned case can be easily achieved using the Mule Requester module, simply by placing the starting point of the flow (the HTTP inbound endpoint in our example) followed by the requester:

Some of its more common use cases are:

  • Load a file in the middle of a flow for processing.
  • Consume messages (one, N, all) from a queue in the middle of a flow.
  • Pull messages from a mail server on demand, to use its data in an enricher for example.

Why releasing it as a Mule module?

  • It’s reusable
  • It’s simple
  • It can be easily installed in MuleStudio
  • It can be used with Maven

Following the first example, let’s show a more complex one: consuming all the messages from a queue on demand. Let’s assume that the use case is to get all those messages after a user called an inbound HTTP endpoint (it could be any other kind of Mule endpoint, e.g. Quartz).

First we have the flow with the starting entry point:

This flow simply receives a request at http://localhost:8081/emptyqueue and delegates to another flow called QueueRequester:

This flow does all the magic:

  1. First it requests a message from the queue ‘input’.
  2. If there is a message, it processes it (just logs the payload in the example) and calls the same flow again, to allow the processing of the next messages in queue.
  3. Otherwise, if there is no pending message in the queue, it just logs the queue has been emptied and finishes.

The full code of this example can be found here:
https://github.com/mulesoft/mule-module-requester/blob/master/mulerequesterdemo/src/main/app/MuleRequesterDemo.xml

That configuration also contains another example that shows how to consume a file based on an expression after calling an HTTP inbound endpoint.

Key feature of the module:

  • Expressions support
  • Possibility of throwing an exception if the requested resource is null
  • Automatic transformer: the return type of the requester resource can be set to any type and Mule will automatically try to transform it

Documentation of its usage and installation details can also be found at its Github main page:
https://github.com/mulesoft/mule-module-requester


We'd love to hear your opinion on this post


27 Responses to “Introducing the Mule Requester Module”

  1. Great, this is a much welcome addition, thank you!

    Note that, before this module, the proper way to request messages was to use the Mule Client, which shields you from having to deal with transport specific concerns, as requests are unified behind endpoint URIs.

  2. Wow! this is a really cool feature we were waiting for, and as said by David, this saves a lot of java coding

  3. Looks great. I have a flow whereby I am sending a request to an http outbound on success will display a json payload to the client. At the same time I also need to send the client a custom email stating the request was successful and provide details. My issue is after I get the http response I do a custom transformer and send to SMTP endpoint. But the smtp endpoint still sends the json payload. On the other hand if I transform and send it to SMTP first the email is formatted properly but the http response instead of being a json payload is a custom email all garbled. It looks like a common case. What would be the approach I need to take. Somebody suggested to take a look at this blog. My case is 2 responses one realtime http and the other asyc smtp with custom email. This seems a usual use case like ordering from Amazon where I see the results of my oreder request on the web page at the same time I get a custom email. Greatly appreciate If you could offer s hint.

  4. Hello Shankaran,

    I would suggest you to use an enricher for that:
    http://www.mulesoft.org/documentation/display/current/Message+Enricher

    Based on your description, you’re not requesting any resource (for which this new module was created).

    If you do have any more questions, please post your question on our forum:
    http://forum.mulesoft.org/mulesoft

    Regards.

  5. As per your suggestion I posted it in mule forum http://goo.gl/n0f4KP and no help and besides we are a paying customer

  6. Hello,

    In that case, please create a Support Case in the Customer Portal:
    http://www.mulesoft.com/support-login

    Regards.

  7. Sebastian,
    Thanks for the reply. This should be such a common use case. If you buy a product from Amazon we fill out a form and get a response(request/response) realtime and then you would get a custom email also detailing your order and shipment. Surprised there is no real knowledge base on such a common scenario. What would be the best practice in this case. We send an http response to the client and then push the payload to a JMS queue and from the JMS queue would be inbound and send SMTP outbound. Our real ask is to do both HTTP response and SMTP at the same time if possible. The issue is is we need to send a custom email to client which is getting garbled by the http response. Thanks.

    Regards,
    Hari

  8. Where’s the xsd-schema file for this module (mule-mulerequester.xsd). Cant find it anywhere…

  9. Hello Petri,

    The XSD is located inside the JAR file under the META-INF folder.

    • I’m using Anypoint Studio and have problems with the mulerequester schema URL not being correct (i.e. not existing on http://www.mulesoft.org) – now that you’ve told us where the XSD is hidden how do we refer to in xsi:schemaLocation?

      Thanks

      • You need to install the module in Studio. That will automatically add the required JAR as part of the deployment.

  10. How would I use it with FTP without deleting the files on the server?

    Thanks,
    /Vins

  11. You have to do a few things to achieve that (since the deletion is done by the FTP transport itself, not by the requester):

    1. Extend org.mule.transport.ftp.FtpMessageRequester and override postProcess (the method in charge of deleting the file).
    2. Create a custom factory to instantiate your custom FtpMessageRequester
    3. Create an FTP connector and set your custom FtpMessageRequester for it:

    <ftp:connector name="ftpConn" >
         <spring:property name="serviceOverrides">
            <spring:map>
                <spring:entry key="requester.factory" 
                    value="your.package.CustomFactory"></spring:entry>
            </spring:map>
        </spring:property> 
    </ftp:connector> 

    4. If you have more than one FTP connector you have to set it on the requester call:
    resource="ftp://user:password@host:port/some/path?connector=ftpConn"

  12. Thanks Sebastian.

    It worked but with side effect. Since I’m not deleting, it goes infinite loop MuleRequesterModule.requestCollection(…). I’m now very deep in the code trying to basically overwriting everything in FTP package which is bad.

    The current FTP implementation always get just one file even though the FTPClient (apache) is returning all the files.

    Regards,
    /Vins

  13. Yes, that’s expected with the current design of the FTP transport. Unfortunately, the requester relies on the transport handling the resource so it’s limited by this.

    An option would be to use the moveToDirectory if you don’t want to delete the files. I know it’s not the same as keeping them where they originally are but it’s easier than overriding the whole requesting mechanism in FTP.

  14. This is an excellent module, just what I needed….however I am running into a problem I was hoping you could comment on:

    I have implemented the requestor with a FTP connector, and I have been partially successful receiving multiple files, however i lose filenames in the process. Any chance you could take a look at the issue and comment?

    Issue posted on StackOverflow with code: http://stackoverflow.com/questions/31843366/using-mule-requester-and-ftp-loses-originalfilename

    Thanks very much.

    • To request more than 1 resource you need to use the request-collection operation.

      You are not losing the file name, it’s simply not part of the message coming out of the requester because it contains a collection of messages. Each of this messages has its own file name and properties.

      Summarizing it:
      request –> MuleMessage with originalFilename
      request-collection –> MuleMessageCollection without originalFilename but having a collection of MuleMessage with originalFilename

      • Thank you for the clarification, that makes sense.

        I have found that if i ForEach over the payload (thinking it would iterate over each of the messages) i don’t have the inboundProperties I would have if I were to request a single message (which would contain the ‘originalFilename’).

        Is there a trick to accessing the individual messages within the messageCollection?

        Thanks again for any clarification.

        • I think I answered my own question… a Collection Spitter seems to have done the trick.

          Thanks again

      • One more question for you on the request-collection:

        I am using a HTTP listener to trigger the Mule Requester to collect the files via FTP. When I have processed the files completely and the message attempts to respond it throws an exception. Details here:
        http://stackoverflow.com/questions/31886713/mule-how-to-respond-after-receiving-a-request-collection-mule-requester

        How would you suggest handling the response in this case?
        Thanks so much.

  15. Hi,
    I am trying to use it the following way.

    I have 2 wmq connectors in my project, each pointing to different physical servers. When mulerequester is trying to hit the URL mentioned in the “resource”, it fails with the following message. Any suggestion on how to resolve this?

    There are at least 2 connectors matching protocol “wmq”, so the connector to use must be specified on the endpoint using the ‘connector’ property/attribute. Connectors in your configuration that support “wmq” are: wmqConnector, wmqConnector1, (java.lang.IllegalStateException) (org.mule.transport.service.TransportFactoryException). Message payload is of type: String

    My mulerequester config:

    • You need to specify the connector as part of the resource in the requester:
      resource=”someURI?connector=

      E.g.:
      file:///some/path?connector=myFileConnector

  16. Hello Sebastian:

    I have a question about mule requester. It is posible to read a file located into another windows machine, I mean a have a shared folder ina a remote windows machine and I want to know if it possible to read files in the shared folder from mule using mule requester.
    Thank you very much.

  17. Hi, I have an use case where I have to poll with a certain interval and then retrieve messages from wmq which has specific priority. I am able to use Poll and WMQ, but I am not able to use selector property for wmq using mule requestor module. Please let me know if any one is having an idea on this.

    Regards,
    Gundi.