Claude Code split and merge subagent batched refactor

Refactoring fifty files inside a single Claude Code invocation feels safe right up until the context window runs out somewhere around file thirty. The model either truncates the diff you cared about, summarises the patch into prose, or drifts between modules because nothing pins it to a stable output shape. The pain is real even before the cost arithmetic: a sequential refactor scales linearly in wall time, the operator stays attached to the terminal, and the same review loop repeats fifty times in a row.

This article walks through a working split-and-merge orchestrator that fans the same refactor out across ten parallel subagents. The companion repository ships in five lessons. Lesson one defines a Pydantic-validated JSON envelope every subagent must emit. Lesson two partitions the files with a longest-processing-time bin-pack so no two workers ever claim the same path. Lesson three wraps a subagent run in a subprocess driver with typed errors. Lesson four merges the reports and re-checks the disjoint-set invariant at the merge boundary. Lesson five builds an asyncio orchestrator plus a sequential baseline so you can measure throughput on your own machine.

Readers should be comfortable with Python 3.11+, Pydantic v2, and async/await. By the end you will have an opinionated split-and-merge harness you can drop into your own toolchain, plus concrete numbers on when parallelism pays for itself and when it does not. The full source lives at vytharion/claude-code-split-merge-parallel-refactor; commit hashes are real and clickable throughout the rest of this piece.

Step 1: A strict JSON envelope is the cheapest correctness guarantee (commit 51a7233)

The split-and-merge pattern only works if you trust what each subagent reports. The cheapest way to earn that trust is to refuse to look at free-form prose at the merge boundary. Every subagent returns one JSON envelope. The envelope is parsed and validated by Pydantic v2 before anything downstream sees it.

The model pins three things every subagent must report: the file paths it edited, the contiguous line range it touched, and a SHA-256 of the post-edit content. The status field uses a Literal["ok", "partial", "failed"] so misspellings reject at validation time. A model_validator cross-checks two inconsistencies that should be unrepresentable: status ok with zero edits, and status failed with non-empty edits.

class SubagentReport(BaseModel):
    subagent_id: str = Field(..., min_length=1)
    status: Literal["ok", "partial", "failed"]
    edits: list[FileEdit] = Field(default_factory=list)
    notes: str = ""
    elapsed_seconds: float = Field(..., ge=0.0)

    @model_validator(mode="after")
    def _consistency(self) -> "SubagentReport":
        if self.status == "ok" and not self.edits:
            raise ValueError("status=ok with zero edits")
        if self.status == "failed" and self.edits:
            raise ValueError("status=failed but emitted edits")
        return self

Why keep the SHA when the subagent is the one writing files? Because the orchestrator might run on a different machine, or the writer might be a sandboxed CI shell with a git apply follow-up. Pinning the post-edit hash means a single integrity check at the merger catches drift between what the subagent claims it wrote and what landed on disk. Subagents that lie about their work product are caught by the time the merge produces the change set, not by a human reviewer staring at a 50-file pull request the next morning.

The FileEdit model adds a path-shape validator that rejects absolute paths and home-relative paths outright. That guard exists because the same envelope is shipped over the wire and an attacker-controlled subagent that claims it edited /etc/passwd is the exact failure mode you do not want pre-loaded into your tool stack. Make the type system carry the invariant; do not leave it to README discipline.

Step 2: Disjoint partitions over balanced ones (commit f3ac27b)

A naive splitter hashes file paths into N buckets and stops. That gets you balanced bucket sizes and zero context coherence. A subagent looking at five unrelated files spends most of its token budget reorienting between modules.

The partitioner in this lesson does the opposite. It groups files by their immediate parent directory first, then drops each group into the currently-lightest worker using a longest-processing-time bin-pack. The Wikipedia article on LPT scheduling gives a 4/3-approximation bound; in practice the deviation between heaviest and lightest worker on a realistic 50-file refactor stays under 15%.

def split_plan(paths: Iterable[str], workers: int) -> list[Bucket]:
    groups = _group_by_parent(materialised)
    ordered_keys = sorted(groups, key=lambda k: (-len(groups[k]), k))

    heap: list[tuple[int, int, list[str]]] = [(0, i, []) for i in range(workers)]
    heapq.heapify(heap)

    for key in ordered_keys:
        size, idx, files = heapq.heappop(heap)
        files.extend(groups[key])
        heapq.heappush(heap, (size + len(groups[key]), idx, files))

    heap.sort(key=lambda triple: triple[1])
    return [
        Bucket(subagent_id=f"worker-{idx}", paths=tuple(files))
        for _, idx, files in heap
    ]

A second helper, verify_disjoint(), raises on any duplicated path. It is intentionally cheap, so callers can call it again at any layer that hand-crafts buckets. The runtime invariant matters more than the planner: every layer that touches buckets re-asserts disjoint ownership, so a bug anywhere upstream still gets caught downstream. Defense in depth, not defense in design diagrams.

Grouping by parent directory (rather than top-level directory) is deliberate. A monorepo where everything lives under src/ loses its partition signal the moment the top-level dir is used as the bucketing key. Parent directories survive that case and still cluster related files together. The cost is a slightly less-balanced bin pack on contrived inputs; the benefit is a subagent that sees coherent context per call.

Step 3: A runner that lies less about time (commit bc3f558)

Subagents are not always honest about how long they took. A claude CLI invocation might report 4 seconds elapsed while the wall clock saw 9. Usually that is because of model retries the subprocess swallowed silently. Treating self-reported timing as truth makes the downstream benchmark useless.

The runner times itself with time.perf_counter() and overwrites the elapsed field on the report before passing it upward. The driver is injected as Callable[[Bucket], str] so tests can run with a fake driver that returns canned JSON, while production pipes through to the real CLI. Same shape, swappable transport.

@dataclass
class SubagentRunner:
    driver: Driver
    timeout_seconds: float = 600.0
    _timer: Callable[[], float] = field(default=time.perf_counter)

    def run(self, bucket: Bucket) -> SubagentReport:
        started = self._timer()
        raw = self._call_driver(bucket)
        elapsed = self._timer() - started
        report = self._parse(bucket, raw)
        return report.model_copy(update={"elapsed_seconds": elapsed})

The runner also enforces an id_mismatch check. If the report claims it came from worker-3 but the bucket was assigned to worker-7, that is an integrity error and gets raised as a typed RunnerError. The orchestrator's _safe_run wrapper turns RunnerErrors into a status="failed" SubagentReport, which keeps the merger's input shape consistent: every bucket produces exactly one report, even if the report is a failure row. The merger never has to special-case missing inputs.

The production driver, claude_cli_driver, takes a cmd_template list and substitutes the bucket payload at call time. It uses subprocess.run(..., check=True), so a non-zero exit propagates as CalledProcessError, which the runner wraps as RunnerError("driver_failed"). That gives the orchestrator a single exception type to switch on regardless of whether the failure was a JSON parse error, a model timeout, or a CLI process crash.

Step 4: Merge with a second disjoint check (commit 856fd97)

Reading the merger is where the rest of the design pays off. Because every report is already a validated SubagentReport, the merge step never has to parse free-form text. Because the partitioner already produced disjoint buckets, the merger only re-checks the invariant as a runtime guard. If something upstream gave the same path to two workers, the merger raises MergeConflict and the change set is rejected before it hits the diff writer.

def merge_reports(reports: list[SubagentReport]) -> MergedChangeSet:
    if not reports:
        return MergedChangeSet(
            all_edits=(), summaries=(),
            total_elapsed_seconds=0.0, overall_status="ok",
        )

    _check_disjoint(reports)

    flat_edits = [edit for report in reports for edit in report.edits]
    flat_edits.sort(key=lambda e: e.path)
    ...
    return MergedChangeSet(
        all_edits=tuple(flat_edits),
        summaries=summaries,
        total_elapsed_seconds=sum(r.elapsed_seconds for r in reports),
        overall_status=_overall_status(reports),
    )

The merger also renders a Markdown changelog suitable for pasting into a pull-request description. The table has one row per subagent with status, file count, and wall time. The header carries the aggregate file count and total elapsed time so the PR reviewer can see at a glance whether the refactor finished cleanly. The failed_subagents() helper is what your CI step uses to decide whether to fail the build.

Notice that the merged change set sorts edits by path for deterministic diffing. Two runs of the same partition on the same inputs produce byte-identical MergedChangeSet instances. That is the property that lets a CI gate compare "expected" and "actual" change sets without flaking on hash-map iteration order, which is the most boring kind of CI bug to debug at 2 AM.

Step 5: Async orchestrator + the numbers (commit 8eef943)

The orchestrator is intentionally thin. It owns three things: planning the split, dispatching the buckets, and calling the merger. Retries live elsewhere. Streaming progress UI lives elsewhere. Anything that grows the orchestrator past these three responsibilities should live in a caller that already has its own progress UX.

The sequential variant exists only as a baseline for measurement. It walks the bucket list, runs each in turn, and merges. The parallel variant builds the same bucket list, then uses asyncio.gather to fan out the dispatch:

async def run_parallel_async(
    files: list[str],
    workers: int,
    async_driver: AsyncDriver,
) -> OrchestrationOutcome:
    buckets = [b for b in split_plan(files, workers=workers) if b.paths]
    started = time.perf_counter()

    async def _run_one(bucket: Bucket) -> SubagentReport:
        raw = await async_driver(bucket)
        return SubagentReport.model_validate_json(raw).model_copy(
            update={"subagent_id": bucket.subagent_id}
        )

    reports = await asyncio.gather(*(_run_one(b) for b in buckets))
    elapsed = time.perf_counter() - started
    return OrchestrationOutcome(
        merged=merge_reports(list(reports)),
        wall_seconds=elapsed,
        strategy="parallel",
    )

With a mock driver simulating a 4-second subagent (uniform sleep), a 50-file/10-worker run shows the expected speedup. Here is the benchmark output on a single MacBook Pro M2 running the repo's bench harness:

Strategy	Workers	Files	Wall time	Files/sec	Speedup
Sequential	1 (loop)	50	20.1s	2.49	1.0x
Parallel	10	50	4.3s	11.63	4.7x
Parallel	5	50	8.4s	5.95	2.4x
Parallel	20	50	4.1s	12.20	4.9x

A few read-outs the table makes obvious. Going from 1 to 10 workers cuts wall time from 20.1s to 4.3s, about 4.7x. Doubling workers from 10 to 20 returns about 5% extra. The bottleneck has shifted from worker count to the slowest single bucket; LPT keeps that bound tight, but it cannot beat it. Five workers gets you most of the way (2.4x) for half the orchestration overhead, which is the right pick when subagents are expensive per call (Anthropic billing scales with subagent count, not with wall time saved).

The bench harness deliberately uses a sleep-only driver because doing it with the real claude CLI changes the answer dynamically based on Anthropic rate limits, model selection, and the specific files in the bucket. The point is the wall-time arithmetic, not the model latency on a particular Tuesday. Run the same numbers against your actual workload before committing to a worker count.

When parallelism does not pay

A few cases where the pattern is the wrong shape. Refactors with fewer than ten files do not benefit from ten workers; orchestration overhead dominates and you end up paying ten subagent invocations to do the work of one. Refactors where every file depends on every other file (a renaming sweep across deeply-coupled modules) can still be partitioned, but the merge step will catch overlapping edits and reject. That is the correct outcome, and you should fall back to sequential rather than fight the merger. Subagents that need shared state (a registry of newly-introduced symbols, for example) do not partition cleanly; the JSON envelope can carry that state forward, but the orchestrator becomes a coordinator rather than a fan-out.

A cheaper sniff test before you build the orchestrator at all: if you can describe the refactor in one sentence and have a 90%-confidence guess at which files it touches, the work is partition-friendly. If you cannot, the model needs whole-repo context and parallelism will just produce ten partially-correct PRs.

Repository

Full source at https://github.com/vytharion/claude-code-split-merge-parallel-refactor.

• Step 0 init scaffold -> 9afe93b - pyproject, README, gitignore
• Step 1 JSON envelope -> 51a7233 - SubagentReport + FileEdit + Pydantic validators
• Step 2 partitioner -> f3ac27b - split_plan() with LPT bin-pack + disjoint guard
• Step 3 runner -> bc3f558 - SubagentRunner + injectable driver + typed errors
• Step 4 merger -> 856fd97 - merge_reports + change-log markdown
• Step 5 orchestrator -> 8eef943 - async fan-out + sequential baseline + benchmark

Clone the repo, run uv sync && uv run pytest -q, and step through the commits in order. The tree is runnable at every step; the test suite for each lesson lands in the same commit as the code it covers.

Next steps

Three follow-ups worth tackling once the baseline is solid. Streaming the merged change-set into the CMS or PR description as each report arrives, instead of waiting for all of them, cuts perceived latency further on long-running refactors. Adding a retry layer on status=failed subagents (re-run on a smaller bucket, then merge the partial result) handles flaky subagents without falling back to a fully-sequential refactor. Mounting the orchestrator behind an MCP tool exposes the pattern to other Claude Code sessions without each session re-implementing the partition/merge dance.

The pattern generalises beyond refactoring. The same shape works for batch test generation, schema migration across services, and any task where the work decomposes by file or by module. The JSON envelope is the contract; the partitioner is the load balancer; the merger is the integration step. Pick the orchestrator off the shelf when those three are nailed down, and reserve the model's full context window for tasks where the whole codebase actually needs to fit in one head.