A demo shows an AI agent can build a Mule integration once. The question that matters for an enterprise team is whether it builds the right one, correctly, every time a developer asks. Hence, we introduced the Build Mule Flow Integration Skill; a skill that turns plain-language requests and specs into a working, buildable Mule application by grounding every design decision in live connector metadata. We tested 74 real-world integration tasks, ran each task three times (k=3), on two platforms, and compared agent performance with and without the skill.
The result: When the skill is introduced, the agent scores 0.95–0.97 out of 1.0 and builds a deployable application nearly every time. Without it, the same model scores 0.40–0.56 and fails to produce a working build on most attempts. The difference comes down to a few habits — gathering integration requirements, reasoning with and checking live connector metadata, and running proper validations with error handling — which happens in almost every run with the skill, and in zero runs without it.
The skill: check the catalog before you write the flow
The Build Mule Flow Integration Skill turns a plain-language request (“sync new Salesforce accounts to a Slack channel”) into a working Mule application. Rather than generating XML and hoping it compiles, the skill imposes the discipline a careful developer would bring to the same task, in two distinct phases with a decision point in between:
Phase 1 — Design. The skill identifies the systems involved, searches Anypoint Exchange for the right connectors, and pulls their actual current metadata — connection config, real operations — rather than recalling a connector from training data. Connector details drift over time: version numbers get retired, operations get renamed. For example, when a request calls for sending a message and waiting for a synchronous acknowledgment, the skill selects the operation from the retrieved contract whose behavior actually matches, with the configuration that operation requires. In contrast, an unaided model writes from memory and is often wrong in ways that look plausible. The skill presents a Technical Design Summary for developer approval before writing a single file.
Phase 2 — Build and verify. Only after approval does it generate the project, using the connector versions, operations, configurations and the application logic pinned from the design. It’s not done when the XML is written — it’s done when the mvn clean package returns BUILD SUCCESS. If a build fails, the skill treats the compiler error as a pointer back into the connector metadata, not something to paper over.
The 2-phased approach is what lets most of mule projects compile on the first build attempt and nearly all of them reach a clean, deployable build, and it is most of what separates the agent with skill from an unaided one.
How we know the score is trustworthy
Every run is graded automatically against a task-specific contract (not one global rubric), weighted across three criteria: plan quality (20%, judged by an LLM against a reference plan), tool use (30%, deterministic rules), and the generated artifact itself (50%, a mix of structural checks and LLM judgment). Two pass/fail gates ride alongside: did the project build, and did the agent ask the right clarifying questions.
The benchmark evaluates the skill rather than the model underneath it. The evaluation framework runs in two stages. First, an agent is given a task and left to work, while everything it does is captured: the reasoning it shows, every tool it calls, and the project files it produces. Then a separate scoring engine grades that record against a contract written specifically for that task. The agent that builds and the engine that grades never share state, which maintains independence between execution and evaluation.
Capability vs. consistency — and why the second one is the real test
Agents and underlying LLMs are inherently stochastic. A single successful run tells you an agent can do the job. It doesn’t tell you it will do the job again next time — an agent built on a LLM inherits its variance. This distinction divides a capable assistant from a dependable one, so we ran every task ran times, and we tracked two different questions:
- Pass@3 — capability: across k attempts, did it succeed at least once in k tries?
- Pass^3 — consistency: across all k attempts, did it succeed on all k tries?
Pass@k rewards the best case; Pass^k demands every case. As runs are added (as k increases), the two measures might naturally pull apart: more attempts raise the probability of at least one success, so Pass@k climbs toward 100%, while every additional run is another opportunity for failure, so Pass^k falls. The gap between the two is the measurable cost of inconsistency.
The same skill, measured on two platforms
A developer might work with the integration agent inside their IDE through MuleSoft Vibes, or drive it from the command line through Claude Code. If the skill encodes good practice, it should pay off in both places. The benchmark and the skill are therefore platform-agnostic by design.
The experiment holds three things constant across both platforms: the skill itself, the benchmark tasks, and the scoring contracts that decide success. The sole variable is where the agent executes.
Here are results when we ran the identical skill against the identical tasks on each platform, on the same underlying model in both:
| Metric | Claude Code | MuleSoft Vibes |
| Capability (Pass@3) | ~99% | ~99% |
| Consistency (Pass^3) | ~94% | ~94% |
| Eventual build success | 100% | ~99% |
| Composite score | ~94/100 | ~92/100 |
On both platforms, essentially every run eventually produced a deployable application, with eventual build success reaching roughly 99% on MuleSoft Vibes and 100% on Claude Code. Most runs also built cleanly on the very first mvn package, without any corrective pass, which is the signature of an agent that grounded its design in real metadata before writing a line rather than reaching a build only after repeated fixes.
Capability alone would have reported “basically perfect.” Consistency is what shows there’s still a few points of variance worth chasing — and where the production-quality skill is defined.
The unaided-model comparison is where it gets stark
Every number and result so far describes the agent with the skill. With the unaided model comparison, the control condition isolates the workflow’s contribution from the underlying model and tools: the same benchmark tasks, on the same two platforms, scored by the same contracts, with the skill switched off. The agent kept its general coding ability and its tools; it lost the workflow that tells it to investigate the catalog, design before building, and verify by compiling.
The clearest behavioral difference can be observed both in the final scores and in what the agent did. Turn the skill off, keep the same model and tools, run the same 74 tasks:
- Composite score: 0.40 (Claude Code) and 0.56 (MuleSoft Vibes), vs. 0.95–0.97 with the skill.
- Build success on matched tasks: 0 of 5 runs compiled on Claude Code, 3 of 5 on MuleSoft Vibes — vs. essentially all runs with the skill.
- Connector metadata lookups: zero, across every unaided run. The model was designed from memory and improvised configurations instead.
Without the skill, the agent designs requirements from internal knowledge, hallucinates connector configurations, and in some cases selects tools without evidence that they matched the requested connector contract. With the skill, the agent grounds the design based on the live connector catalog in almost every run; without it, that grounding does not occur at all. Everything downstream (the right version, the right operation, the clean first maven build) follows from that one habit, and its absence is why the unaided results look the way they do.
Bottom line
Run once, most agents look impressive. Run 74 tasks three times each, and the gap between “worked in the demo” and “works every time” becomes measurable — and for this skill, that gap is narrow: ~99% capability, ~94% consistency, versus an unaided model that rarely produces a compiling build at all. The causal mechanism is the metadata-first workflow: the agent grounds each design in the real connector catalog before writing a line, which is exactly the step an unaided model skips. For a developer choosing between the two, the practical takeaway is direct: the skill is the difference between an agent you have to check and one you can hand a task to.
To learn more about this Skill check out the documentation or the MuleSoft Blog here.
