There is no unique or simple answer to that question, because it depends on the use case. But there is definitively a place for ESB and a place for BPM and they can and should leverage each other. Let’s take a look at how to answer the question above.
At a high level, ESB and BPM are separate but interdependent.
In another words, ESB solves the problem of complex system integration (usually stateless, short-lived transactions), while BPM solves the problem of modeling and orchestrating business processes, integrating people and systems (usually stateful, long-running transactions).
You can think of SOA as a 3 layer architecture, the lowest being “service-enablement” or exposing existing systems as services, the middle being “service orchestration” allowing to create and compose coarse-grained “business services”, and the highest being “process orchestration” to create business processes.
Normally ESB is assigned to the middle layer — orchestrating low-level services into larger service units, which will be exposed to the business for use in processes — and BPMs in the top layer. Mule not only provides value on the middle layer, but it is also a container for component and services, providing “service-enablement” therefore adding value on the bottom layer as well.
At the end of the day, is not when to use one or the other, but rather why or how make them work together. In another words to be successful with business processes first you need to have all your systems and apps exposed as services; that is where Mule ESB comes into play.
The other way around is another important use case too. Many of our customers are using Mule ESB to expose a process as a service or to kick off a process instance.