Keep your data with the new Mule 4 ObjectStore connector

An ObjectStore is a Mule component which allows for simple key-value storage. Although it can serve a wide variety of use cases, it’s mainly designed for:

• Storing synchronization information like watermarks
• Storing temporal information like access tokens
• Storing user information

If you are a Mule 3 user, then you’re already familiar with this concept. However, we feel that creating and defining these stores could be made much simpler. For example, it shouldn’t require the use of a Spring bean.

In Mule 4, we have introduced the new ObjectStore connector that solves the aforementioned limitations with ease and power, as well as includes a new capability to define custom stores you can later reference.

Let’s start with the basics

You can store single values:

<os:contains id="#[userId]" objectStore="tokensStore" />

You can also use it to store complex structures you can build inline:

<os:store key="state">
	<os:value>
    <![CDATA[#[
       output application/json
       ---
       { "id": attributes.queryParams.id,
          "timestamp": now(),
          "name": payload.name
          }
      ]]]>
    </os:value>
</os:store>

The store operation can be used either for storing new values or updating existing ones, depending on the value of the failIfPresent parameter. When that parameter is set to false (default value) then any pre-existing value associated with that key will be overwritten. If the parameter is set to true, then an OS:KEY_ALREADY_EXISTS error will be thrown instead.

Another important consideration is regarding null values. It is not possible to store a null value. However, a common use case is to:

• Obtain a value (most likely by evaluating an expression or transformation)
• Testing the value for not null
• Storing it if present
• Do nothing otherwise.

The failOnNullValue parameter simplifies this use case. On its default value of true, an OS:NULL_VALUE error is thrown if a null value is supplied. However, when set to false, a null value will cause this operation to do nothing: no error will be raised but no value will be altered either.

Retrieving information from the ObjectStore

You can retrieve single values like this:

<os:retrieve key="userId" />

You can also provide default values when the key is not present in the store:

<os:retrieve key="timestamp">
	<os:default-value>#[now()]</os:default-value>
</os:retrieve>

You can even have default complex structures:

<os:retrieve key="state">
	<os:default-value>
      <![CDATA[#[
         output application/json
         ---
         { "id": attributes.queryParams.id,
            "timestamp": now(),
            "name": payload.name
            }
      ]]]>
    </os:default-value>
</os:retrieve>

If the parameter was not provided or resolved to a null value, then an OS:KEY_NOT_FOUND error will be thrown. Otherwise, the defaultValue will be returned. But keep in mind that such value will not be stored.

Note: All these operations are synchronized on the key level. No other operation will be able to access the same key, on the same object store while this operation is running. If Mule is running in cluster mode, this synchronization is also guaranteed across nodes.

Defining your own stores

Scopes of ObjectStores

Each Mule application has its own default object store, which is a persistent one, always available to that app. However, quite often you’ll find yourself in the need to define additional object stores. Reasons for doing that can be:

• You want to partition your information by storing them in different stores
• You want to use advanced ObjectStore features such as:
  • Transient/Persistent storage
  • Specify a time to live (TTL)
  • Specify a max capacity
• You want to keep different components from sharing state by feeding them with different stores
• You want different components to share information by feeding them with the same store

You can define an object store globally in the application so it can be referenced by name and shared between multiple components. Or you can create an object store which is “private” to a particular component.

Global object stores are defined as top-level elements and are given a name through which other components can reference them. For example, let’s suppose we want to create a store for access tokens:

<os:object-store name="tokensStore"
	 entryTtl="1"
	 entryTtlUnit="HOURS"
	 maxEntries="100"
	 persistent="true"
	 expirationInterval="30"
	 expirationIntervalUnit="MINUTES" />

Here we have a store which:

Is persistent: Values are stored on the disk and can survive a system restart. Setting persistent to false will result in a transient store which only stores information in memory.

• Is expirable: There will be an expiration thread that runs every 30 minutes and discards the elements that have exceeded their TTL or have exceeded the maxEntries limit
• Has a 1 hour TTL: Because these tokens are highly sensitive, we don’t want to have them for more than an hour. Every entry older than that will be automatically deleted
• Has a size limit: If the store grows to more than 100 entries, the exceeding items will be discarded when the expiration thread runs.

Note: This is just an example. We don’t necessarily recommend that an ObjectStore holding access tokens should be configured in this way.

Now that we have our store, let’s configure the Salesforce connector to use it:

<sfdc:config-with-oauth name="salesforce-oauth"
  consumerKey="${salesforce.consumerKey}"
  consumerSecret="${salesforce.consumerSecret}">
          <sfdc:oauth-callback-config domain="localhost" localPort="8082"
                remotePort="8082" path="callback" connector-ref="HTTP_HTTPS" />
          <sfdc:oauth-store-config objectStore="tokensStore" />
</sfdc:config-with-oauth>

The configuration above shows a Salesforce connector authentication through OAuth, and also shows the storing of the tokens in the store we just created.

Use a global ObjectStore when:

• You need to share state across components. Because the store can be referenced by name, other components can also use it. Suppose you have a second instance of the Salesforce connector, but with a slightly different configuration. You can use this other instance using the same OAuth tokens, avoiding the need to authenticate again.
• You need to share state across Cluster Nodes. When you’re using Mule in cluster mode, you may want to have information which is available to every node in the cluster. Global ObjectStores are ideal for that.
• You need to use the information in your application’s logic. The ObjectStore connector is not only capable of defining that store, but also manipulating it. All of the modules’ operations operate on the application’s Object Store by default. However, these operations accept referencing a global object store as well. For example, suppose you want to have an audit flow which verifies if certain users are currently authorized. You could do something like this:

<os:contains id=”#[userId]” objectStore=”tokensStore” />

Private stores

Another alternative is to define a private object store; one which is not defined as a global element and doesn’t have a referable name. For example:

<idempotent-message-validator idExpression="#[payload]" valueExpression="#[payload]">
	<os:private-object-store
	                         entryTtl="20"
	                         entryTtlUnit="MILLISECONDS"
	                         maxEntries="20"
	                         persistent="false"
	                         expirationInterval="20"
	                         expirationIntervalUnit="MILLISECONDS"/>
</idempotent-message-validator>

In this example, we gave the idempotent message validator a custom store which only it can access.

Use a private ObjectStore when:

• To share state can either be a feature or a security risk.
• To be able to manipulate the store from the connector can either be a feature, or something error-prone. Some developer could accidentally end up doing a bad use of the clear operation and delete all the authorization data

Limitations

Finally, some words on ObjectStore limitations. This is not a universal solution for data storage. It doesn’t replace an actual Database and it’s not suitable for every use case. Most importantly, it does not support transactional access or modification. For use cases in which ACID semantics are needed, or cases in which you expect the same key being updated in parallel, consider another solution.

Thank you for reading and please send us feedback!