On June 30, 2026, IBM Research put a sharper edge on a question enterprise teams have been circling for months: if an AI agent can rewrite Java code, can it also migrate a real application without breaking the system around it?
ScarfBench, short for Self-Contained Application Refactoring Benchmark, is an open benchmark designed to evaluate AI agents on enterprise Java framework migration across Spring, Jakarta EE, and Quarkus. The important change is not just that the benchmark exists, but what it treats as success. ScarfBench does not stop at code similarity or syntactic translation. It asks whether the migrated application builds, deploys, and behaves correctly at runtime.
That distinction matters because enterprise Java migration is not a line-by-line translation problem. It is a systems problem. Build files change. Framework conventions shift. Dependency graphs get rewritten. Runtime behavior has to survive the move. For AI tooling vendors and platform teams, ScarfBench moves the benchmark target from generating plausible code to preserving the operational contract that enterprise software actually depends on.
Translation is no longer enough
Traditional code-generation benchmarks have helped demonstrate that models can produce syntactically valid fixes, boilerplate, and partial refactors. ScarfBench is aimed at a harder class of work: cross-framework migration in applications where correctness is defined by how the system behaves after the refactor, not how close the output looks to the input.
That framing forces a more disciplined evaluation model. An agent that rewrites annotations but leaves the build broken has not succeeded. Neither has one that compiles but fails once dependencies, runtime wiring, or framework-specific assumptions come into play. In enterprise Java, those failures are not edge cases; they are the migration work.
ScarfBench’s open design also matters. Because the benchmark is public, it can become a shared reference point for what “good” looks like in AI-assisted modernization. That is especially important in a market where vendors often showcase isolated demos or narrow productivity gains. A benchmark that checks build success, deployment readiness, and behavior can expose the gap between a tool that generates code and a tool that can actually carry an application across framework boundaries.
What behavioral correctness really means
Behavioral correctness is the part of migration that code similarity hides.
In enterprise Java, applications often depend on framework lifecycle hooks, configuration conventions, injection patterns, and library interactions that do not show up in a superficial diff. A migration can preserve method names and class structure while still altering execution order, response handling, transactional behavior, or error semantics. If an AI agent does not understand that full path from source to runtime, it may produce a version of the application that looks reasonable and still fails in practice.
That is why ScarfBench’s emphasis on end-to-end viability is technically consequential. The benchmark implies that agents need to do more than map APIs from one framework to another. They have to reason across build systems, adapt project scaffolding, manage cross-framework dependencies, and preserve the behavior that users and downstream services actually observe.
For tooling vendors, that widens the product requirement. Migration assistants now have to be evaluated as systems tools, not autocomplete layers. The relevant question is no longer whether a model can draft a migration patch, but whether it can complete a chain of tasks that includes compiling the project, deploying it successfully, and validating runtime outcomes against expected behavior.
That is a much higher bar, and in enterprise environments, it is the bar that matters.
Why an open benchmark changes the market
An open benchmark does more than standardize measurement. It reshapes procurement language.
If ScarfBench gains traction, vendors will have to explain migration claims in terms that technical buyers can test. That can accelerate adoption for products that already handle dependency-aware refactoring, build repair, and runtime validation. It can also slow buyers down where the tooling is less mature, because the benchmark exposes what a demo may conceal: failures at compile time, packaging time, or runtime.
For enterprise customers, that is a useful correction. Migration projects are costly precisely because they are full of hidden coupling. A benchmark that captures those couplings gives teams a more credible basis for comparing agents, assistants, and orchestration layers. It also encourages a shift in platform strategy: away from generic code generation and toward workflow-aware automation that can be inserted into existing engineering controls.
The likely competitive split is straightforward. Vendors that can connect language models to project-aware reasoning, build validation, and behavior checks will be better positioned. Products that can only produce text will look increasingly weak in a benchmark built around execution, not prose.
What teams should do now
Teams evaluating AI-assisted modernization should not wait for a vendor to declare readiness. They should build their own internal evaluation loops around the same kinds of criteria ScarfBench uses.
That means three practical steps.
First, treat migration as an executable pipeline, not a code review exercise. Any AI-assisted refactor should be checked for build stability before it is allowed into a broader rollout path.
Second, add deployment and runtime verification to the evaluation loop. If the migrated service does not start cleanly, does not pass integration checks, or changes observed behavior, the migration is incomplete.
Third, instrument the cross-framework dependencies that tend to fail silently. Configuration, lifecycle behavior, library compatibility, and service interactions should all be part of the test harness. In enterprise Java, these are the failure modes that separate a useful migration assistant from a glorified code generator.
For organizations already experimenting with developer tooling, this also argues for tighter CI/CD integration. ScarfBench’s criteria map naturally to automated gates: compile, deploy, run, compare behavior. If an AI agent cannot pass those gates in a controlled environment, it should not be trusted with production migration work.
A trend signal with real operational consequences
The June 30 coverage window is notable because it shows where the conversation is moving. AI-enabled developer tooling is no longer being judged only by how much code it can emit. It is being judged by whether it can preserve meaning in large, messy systems.
That shift does not mean AI is ready to modernize enterprise estates on its own. It means the field is maturing toward a more credible standard. Benchmarks like ScarfBench make that maturity measurable by tying success to behavior, deployment, and runtime correctness rather than to surface-level similarity.
For technical teams, the implication is simple: if migration work is on the roadmap, evaluation has to move upstream now. Governance, instrumentation, and realistic risk assessment will matter as much as model quality. ScarfBench does not settle the question of enterprise Java modernization, but it makes the right question harder to avoid: can an AI agent preserve the application, or only rewrite it?
