Zum Inhalt springen
Repo-Strategie

Splitting a monorepo without losing the history

Splitting a repository is not a tooling question. The tool is understood in one afternoon. The hard part is the decision before it and the cleanup after it, and both get skipped regularly, because the cut itself goes so fast.

This post is a full tutorial and assumes nothing beyond Git on the command line. It takes a tool online shop that has grown inside a single repository, brings about the moment when the shared repository no longer carries, performs the split step by step with the history preserved, and then shows what goes wrong on split day anyway. At the end comes the honest counter-proposal: when you are better off not splitting at all.

Contents

The scene: one shop, one repository

An online shop sells tools: drills, jigsaws, cordless screwdrivers, plus accessories and spare parts. It started as one application and grew into five services. To this day everything sits in one repository.

werkzeugshop/
├── storefront/          web front end
├── catalog-service/     articles, categories, search
├── checkout-service/    cart, order, payment
├── pricing-service/     list prices, promotions, quantity tiers
├── shared-contracts/    shared DTOs and OpenAPI files
└── infra/               Terraform, Helm charts, pipelines

One clone, one checkout, one pipeline. Whoever changes an interface changes every caller in the same commit. The build is green or red, and both apply to the whole shop.

Why that was the right call for a long time

There is no version matrix in this setup, because there is nothing to version. What sits on the main branch fits together. That is the biggest advantage of a shared repository, and it is usually missed only once it is gone.

An example from the shop. The pricing engine has so far delivered prices as an integer in cents. For spare parts with tiny prices that is supposed to become a decimal value. In the shared repository this is one change:

// shared-contracts/src/main/java/shop/contracts/PriceDto.java
public record PriceDto(String sku, BigDecimal amount, String currency) { }

The same commit pulls pricing-service and checkout-service along. No release of the contract library, no coordination, no transition phase with two valid formats. One review, one green build, done.

This property does not disappear when the repository grows. It only gets overtaken by other costs at some point.

The trigger: someone from outside needs the prices

The shop wants to introduce quantity tiers for spare parts. One price from twelve units, another from forty-eight, plus special rules for trade customers. The business logic for it is more extensive than expected, and nobody on the team is free. So an external contractor comes in who does exactly this rebuild.

With that, one question is suddenly no longer theoretical: he needs write access to pricing-service. But he is not supposed to see the checkout-service, which holds the payment integration including key management, and certainly not its full history.

Git cannot resolve that. Permissions apply at repository level, not at directory level. There is no setting that releases a subdirectory to a collaborator and hides the rest. That ends the discussion of principle, and not through an architecture argument but through a property of the tool.

That is the most important point at this stage: the split does not come here because somebody finds the structure prettier. It comes because there is a requirement that cannot be met any other way.

The first attempt, which looks harmless

The obvious route takes two minutes. Create a new repository, copy the folder in, commit once.

mkdir ../pricing-service
cp -r werkzeugshop/pricing-service/* ../pricing-service/
cd ../pricing-service
git init
git add .
git commit -m "initial import"

The result works. The service builds, the tests run, the contractor gets access, and nobody sees more than he is supposed to. For the moment the job is done.

What gets lost in the process only shows up later, and that is why this route looks so tempting.

Three weeks later the answer is missing

A customer orders fourteen saw blades and gets the tier price. With eleven blades he does not. The boundary sits at twelve, and somebody asks why twelve, actually, and not ten.

In the old repository the answer would be one command away:

git log -L :calculateTieredPrice:src/main/java/shop/pricing/TierCalculator.java

In the new repository there is exactly one commit, and it is called “initial import”. The discussion in which the twelve came about, the reference to the ticket, the comment from the colleague who pointed at the packaging unit back then: all gone. Not deleted, because it is still in the old repository. But it sits where nobody looks any more, and it hangs on paths that no longer exist in the new repository.

That is exactly what the split with history is about. Not completeness for its own sake, but the ability to answer a question a year from now.

The four signals that justify the split

Before the technical part comes the decision. Four observations carry the split, and in practice they almost always show up together.

Different release cadences. The storefront goes live several times a day. The checkout-service bundles changes into approvals, because payment processing hangs off it. In the shared repository the slower cadence forces its frequency on the faster one, or the team builds branch constructs to decouple the two. When those constructs start needing rules of their own, that is the signal.

Separate responsibility. As long as one team owns everything, every review is a technical question. As soon as two teams are in charge, it turns into a question of who is responsible. Rules about code owners map that, but they do not enforce it.

Access that has to differ. The criterion from our scene. It is the only one of the four that is not negotiable.

Feedback times nobody carries any more. When every change to the storefront builds the checkout along with it, the wait grows with the whole shop instead of with the changed part. Before splitting over that, though, the pipeline belongs on the test bench. It should only build what has changed anyway.

Three reasons that do not hold

Just as important is what does not justify the split.

The repository has supposedly grown too big. Size is a tooling problem, and Git has answers for it, further down. Whoever splits because of the clone time trades an inconvenience for a permanent coordination task.

The folder supposedly looks cluttered. Clarity comes from how the modules are cut. It has nothing to do with the number of remotes. A shop cut badly gets harder to correct once it is spread over five repositories, because the boundary is then written into the infrastructure.

Microservices supposedly need one repository each. That is convention. Being independently deployable has nothing to do with where the source code is stored, and the shop has been proving that for years.

What you need

Git and one additional tool are enough for the steps that follow.

git filter-repo is not part of the Git distribution. It is a single Python script that you put on the search path. According to the project documentation it requires Git from version 2.36.0 and Python 3 from version 3.6.

python3 --version
git --version
git filter-repo --version

If the last command answers with “not a git command”, the script is still missing. It is available from the common package managers and can alternatively be dropped in as a file and made executable.

A word about git filter-branch, which many older guides recommend: the Git documentation now explicitly advises against it. It gives two reasons. The command damages the history in ways you do not notice right away, and it is so slow that reproducing such damage becomes a test of patience. For our case that settles it.

Step 1: determine the boundary

Before the first command comes the question of what exactly moves out. In the tool shop the answer is not simply pricing-service/, because the service uses shared contracts.

grep -rl "shop.contracts" werkzeugshop/pricing-service/src | head

There are three options, and the decision belongs before the split, not after:

  1. The contracts stay in the monorepo and get published as a library from now on. Then the extracted service needs a dependency on a version, and with that the version matrix begins.
  2. The contracts move along. Then they exist twice, and the two copies drift.
  3. The contracts get a repository of their own. Clean, but it is a second split.

For the shop the choice falls on the first variant, because checkout-service and storefront need the same contracts. From now on the extracted service pulls them as a versioned artifact.

Write this decision down before you carry on. It is the part you can no longer reconstruct in six months.

Step 2: create the fresh clone

git filter-repo rewrites history. That happens on a copy which exists for exactly this purpose. The working directory somebody is currently developing in stays untouched.

git clone https://github.com/werkzeugshop/werkzeugshop.git pricing-split
cd pricing-split

The tool brings a precaution for that: if it does not run in a fresh clone, it aborts, unless you override it explicitly. This attitude pays off for the whole migration. A clone that only exists for the attempt anyway is allowed to fail. You delete it and start over, and nobody loses anything.

With a large repository it is worth a look at the clock here. How long the filter run takes is better measured on this copy than found out on migration day.

Step 3: filter out the history

Now the actual cut. The command keeps only one directory and moves it to the root of the repository.

git filter-repo --subdirectory-filter pricing-service

Afterwards the content of the service sits in the root directory, and the history only contains commits that touched this directory. You can check that right away:

ls
git log --oneline | wc -l
git log --oneline | tail -5

The output shows the difference to the copied folder from earlier:

src  pom.xml  README.md  Dockerfile
487
a3f19c2 Pricing engine extracted from the monolith
7d4e881 Promotional prices with validity period
1c9a03e Rounding to packaging unit
0b2e447 Quantity tier from twelve units, ticket SHOP-412
9f1d330 First version of list prices

The answer to the question about the twelve is back, including the ticket number. The tool cleans up afterwards on its own and repacks the repository, nothing further to do for that.

Step 4: take the old paths along

Here lurks the mistake that most often goes unnoticed. Path filters work on the paths as they stand in the history, not on today’s name.

The service in the tool shop was not always called pricing-service. For the first two years it sat under preise/, after that under preis-service/; only since the switch to English identifiers has it been called what it is called today. The command from step 3 knows only the current name. Everything that happened before the last rename drops out silently. There is no warning, the result looks plausible, and the history ends on a date nobody notices.

That is why a look back belongs before the filter run:

git log --follow --name-only --format="%h" -- pricing-service/pom.xml | grep -v "^$" | tail -20

If an earlier path shows up there, you take all variants along and rename them in one go:

git filter-repo 
  --path preise/ 
  --path preis-service/ 
  --path pricing-service/ 
  --path-rename preise/:pricing-service/ 
  --path-rename preis-service/:pricing-service/ 
  --path-rename pricing-service/:

The first three entries determine what is kept. The three renames merge the historical paths and lift the result to the root. After that the history is continuous, and git log --follow also finds commits from the time when the service still had a different name.

Step 5: sort out the tags

The tool shop tagged its releases on the project as a whole: v3.4.0, v3.5.0 and so on. These tags travel along when filtering, and in the new repository they are misleading, because v3.4.0 was never a version of the pricing engine but one of the shop.

Two routes are defensible. Either you rename them while filtering, so their origin stays visible. You append this to the same single run from step 4:

git filter-repo   --path preise/   --path preis-service/   --path pricing-service/   --path-rename preise/:pricing-service/   --path-rename preis-service/:pricing-service/   --path-rename pricing-service/:   --tag-rename "":"shop-"

v3.4.0 becomes shop-v3.4.0 that way, and nobody confuses it later with a version of the service. Or you throw them away afterwards and start with your own counting:

git tag -l | while read -r t; do git tag -d "$t"; done
git tag v1.0.0

For the shop the decision falls on renaming, because the release notes in the operations handbook refer to the old numbers. All that matters is that it is a deliberate decision. Leaving them untouched is the one variant that causes trouble later.

Step 6: fill the new repository

You do not have to deal with the reference to the monorepo, the tool has already removed it. That is its second built-in safety measure, and it is deliberate: without origin nobody can accidentally push the rewritten history into the original repository. Whoever really wants that adds the remote back by hand and then knows what they are doing.

git remote -v
# output is empty, filter-repo removed origin
git remote add origin https://github.com/werkzeugshop/pricing-service.git
git push -u origin main
git push origin --tags

Only after that does the contractor come in, and only on this repository. That was the whole purpose of the exercise, and at this point it is reached.

Step 7: shut down the old directory

The step that most often gets dropped, because the actual work seems done. In the monorepo pricing-service/ is still lying around. It still builds, the tests still run, and in half a year somebody who never heard about the migration changes a line there.

cd ../werkzeugshop
git rm -r pricing-service

A note belongs in that spot, one that answers the question before it is asked:

mkdir pricing-service
cat > pricing-service/README.md <<'NOTE'
This service has moved.

New repository: https://github.com/werkzeugshop/pricing-service
The full history is preserved there, including the time as
preise/ and preis-service/.

Development no longer happens here.
NOTE
git add pricing-service/README.md
git commit -m "Pricing engine extracted, pointer to new repository"

Part of it is adjusting the monorepo’s pipeline so it no longer builds the vanished service, and publishing the contracts as a library, as decided in step 1.

The turning point: what goes wrong on split day anyway

The source code is the easy part. What stood out in the tool shop in the days afterwards had nothing to do with Git any more.

The branch protection rules did not exist in the new repository. In the monorepo every change to the main branch needed two approvals. The new repository was fresh, so nothing applied there, and the contractor’s first commit landed straight on main. Nobody had done anything wrong, the rule had simply never been carried over.

The secrets were missing. The service’s pipeline needed credentials for the artifact registry, and those sat in the monorepo. The first build in the new repository failed, and because the error message only spoke of missing permissions, the search took longer than the whole split.

The dependency updates ran into nothing. The bot that proposes version bumps in the monorepo did not know the new repository. For three months not a single proposal came in, and that only surfaced with a security advisory.

References pointed nowhere. Tickets, documentation and commit messages held links to files in the monorepo. Those paths do not exist there any more.

So everything except the source code belongs on the list for split day: protection rules, approval requirements, secrets, automation, references. Whoever only takes the history along has carried over half of it.

What the split costs permanently

The real costs do not fall on migration day. They fall every month after it, and that is why they get underestimated.

The atomic change is gone. Remember the switch of prices to BigDecimal further up, one commit in the shared repository. After the split it turns into a sequence: first publish the contract library in a new version, then lift the pricing engine onto it, then pull checkout and storefront along. In between, two valid formats exist, and somebody has to decide for how long.

With that, every shared interface needs a version, a compatibility promise and a way to announce breaks. That is the actual price of the split, and it is permanent.

The truth about the overall state disappears. Before, the main branch answered the question of which versions fit together. Afterwards it takes a source of its own for that, in the tool shop an environment description in the GitOps repository that records which version runs where.

The pipeline multiplies. Every repository brings its own configuration and its own secrets. What was maintained once is now maintained several times, and the configurations drift apart from day one if nobody actively keeps them together.

The tools compared

Tool Installation Speed Can do single files Recommendation
git filter-repo separate Python script fast yes the route for this case
git subtree split included in Git noticeably slower no, directories only for a one-off cut on a clean boundary
git filter-branch included in Git very slow yes explicitly advised against by the Git documentation

git subtree split is the pragmatic route when no additional tool may be installed and the boundary already sits cleanly on a directory:

git subtree split --prefix=pricing-service -b pricing-only

That produces a branch with the filtered history, which you push into a new repository. What it cannot do: extract single files and merge historical paths. For the tool shop with its three directory names it is therefore not enough.

The other direction: keeping the monorepo workable

If the only pain is the size, there is a cheaper route than a migration. Git brings two mechanisms for it that can be combined.

A partial clone fetches the file contents only when they are needed, instead of transferring the whole history of every file up front:

git clone --filter=blob:none https://github.com/werkzeugshop/werkzeugshop.git

On top of that, git sparse-checkout limits the working tree to the directories somebody actually works on:

cd werkzeugshop
git sparse-checkout set pricing-service shared-contracts
ls

After that only the two directories plus the files on the top level sit in the working tree. The default cone mode deliberately takes only directories and no arbitrary patterns, which keeps the evaluation fast.

Two restrictions belong with it. The Git documentation still lists git sparse-checkout as experimental, and the sparse index that goes with it is a separate, likewise experimental feature that can confuse external tools. For developer machines that is still usually the cheaper answer than a migration whose follow-up costs are permanent.

What these mechanisms do not solve: the access problem from our scene. A partial clone hides nothing, it only loads later. Whoever must not see the checkout code must not clone the repository.

When the split pays off and when it does not

It pays off when access rights have to differ, because there is no alternative for that. It pays off when two parts have permanently different release cadences and the constructs that map this in the shared repository become a burden themselves. And it pays off when separate teams are in charge and every review turns into a question of who is responsible.

It does not pay off as a cleanup, not because of the clone time, and not because an architecture convention suggests it. In those cases you pay coordination costs for a problem that could have been solved more cheaply.

For the tool shop the answer was clear, but it applied to only one of the five services. Storefront, catalog and checkout still sit together to this day, and there is no reason to change that.

FAQ

Do I lose the history if I copy a folder into a new repository?
Yes, completely. The new repository starts with a single commit. The old history lives on in the original repository, but it hangs there on paths that do not exist in the new one, and in practice it is not found any more.

Why should I not use git filter-branch?
Because the Git documentation explicitly advises against it. It gives two reasons: the command damages history in ways you do not see right away, and it is so slow that the debugging afterwards gets out of proportion.

What happens if the directory used to have a different name?
The path filter knows only the paths you give it. Everything before the rename drops out, without a warning. You have to name all historical paths and merge them via renames.

Do I have to take the tags along?
No, but you have to decide. Tags of the overall project are misleading in the extracted repository, because they name a version that never existed there. Either rename or delete.

Can I undo the split?
The filter run itself yes, by throwing the clone away and starting over. That is exactly what the fresh clone is for. What cannot be turned back are the habits afterwards: published versions, references and everything others have already pointed at the new repository.

Do I need separate repositories for microservices?
No. Being independently deployable is a property of the build and deployment route, not of where the source code is stored. A monorepo can ship five services independently.

How long does the filter run take?
That depends on the number of commits and files. Instead of guessing, you measure it on the fresh clone you are creating anyway. This is the one case where the migration allows a free dry run.

Sources

  • Git documentation on git filter-branch, including the warning and the pointer to the alternative
  • Git documentation on git sparse-checkout, cone mode and the status of the sparse index
  • Project documentation of git filter-repo for the requirements and the behaviour with a fresh clone

All commands and examples in this article are our own and were followed through on the setup described.

Takeaway

The split is not a cleanup. It acknowledges a boundary that had long been there in business terms, and in the tool shop it was the access that made it visible.

If you go that way, take the history along. It costs you one extra work step and answers the question a year from now of why the quantity tier starts at twelve units. And plan for the day after: protection rules, secrets, automation and references do not move along by themselves.

The next concrete step is small. Take a directory you believe should become independent, clone the repository fresh and let the filter run go through once. Within minutes you will see whether the history is continuous or whether a rename is waiting somewhere that nobody knew about any more.

← Back to the blog

Vela Atlas is targeted exclusively at businesses (§ 14 BGB). No sales to consumers.
$ lang DE • EN • ES