What are the best practices for un-submoduling a Git submodule, bringing all the code back into the core repository?
13 Answers
If all you want is to put your submodule code into the main repository, you just need to remove the submodule and re-add the files into the main repo:
git rm --cached submodule_path # delete reference to submodule HEAD (no trailing slash)
git rm .gitmodules # if you have more than one submodules,
# you need to edit this file instead of deleting!
rm -rf submodule_path/.git # make sure you have backup!!
git add --force submodule_path # will add files instead of commit reference
# --force adds files ignored by .gitignore
git commit -m "remove submodule"
If you also want to preserve the history of the submodule, you can do a small trick: “merge” the submodule into the main repository, so that the result will be the same as it was before, except that the submodule files are now in the main repository.
In the main module you will need to do the following:
# Create a 'marker' at the submodule commit being used
cd submodule_path
git switch --create "submodule_merge_marker"
git push --set-upstream origin submodule_merge_marker
# Fetch the submodule commits into the main repository
git remote add submodule_origin git://url/to/submodule/origin
git fetch submodule_origin
# Start a fake merge (won't change any files, won't commit anything)
git merge -s ours --no-commit submodule_origin/submodule_merge_marker
# Do the same as in the first solution
git rm --cached submodule_path # delete reference to submodule HEAD
git rm .gitmodules # if you have more than one submodules,
# you need to edit this file instead of deleting!
rm -rf submodule_path/.git # make sure you have backup!!
git add --force submodule_path # will add files instead of commit reference
# --force adds files ignored by .gitignore
# Commit and cleanup
git commit -m "remove submodule"
git remote rm submodule_origin
The resulting repository will look a bit weird: there will be more than one initial commit. But it won’t cause any problems for Git.
A big advantage of this second solution is that you can still run git blame
or git log
on the files which were originally in submodules. In fact, what happens here is just a renaming of many files inside one repository, and Git should automatically detect this. If you still have problems with git log
, try some options (e.g., --follow
, -M
, -C
) which do better rename and copy detection.
-
3I think I need to do your second method (history preserving) on some git repos I have. Could you please explain which part of the above commands causes the files from the submodule to end up in the subdirectory? Is it that you when you do the merge git brings in the file in the top level directory (with its history) but when you do the git add submodule_path it implicity does a git mv for every file? Commented Dec 20, 2011 at 4:43
-
5Basically, yes. The trick is that git does not store rename operations: instead, it detects them by looking at the parent commits. If there is a file content that was present in the previous commit, but with a different filename, it is considered a rename (or copy). In the steps above,
git merge
ensures that there will be a "previous commit" for every file (at one of the two "sides" of the merge).– gyimCommented Dec 26, 2011 at 13:44 -
6Thanks gyim, I started a project where I thought it made sense to split things into a couple of repositories and link them back together with submodules. But now it seems over engineered and I want to combine them back together without losing my history. Commented Dec 27, 2011 at 23:32
-
5@theduke I also had this problem. It can be fixed by, before following these steps, moving all of the files from your submodules repository into a directory structure with the same path as the repository that you are about to merge into: ie. if your submodule in the main repository is in foo/, in the submodule, perform
mkdir foo && git mv !(foo) foo && git commit
. Commented Oct 11, 2013 at 7:27 -
60Needed to add
--allow-unrelated-histories
to force the merge at the fake merge as I was gettingfatal: refusing to merge unrelated histories
, more here: github.com/git/git/blob/master/Documentation/RelNotes/…– vaskortCommented Aug 15, 2016 at 13:23
I've created a script that will translate a submodule to a simple directory, while retaining all file history. It doesn't suffer from the git log --follow <file>
issues that the other solutions suffer from. It's also a very easy one-line invocation that does all of the work for you. G'luck.
It builds on the excellent work by Lucas Jenß, described in his blog post "Integrating a submodule into the parent repository", but automates the entire process and cleans up a few other corner cases.
The latest code will be maintained with bugfixes on github at https://github.com/jeremysears/scripts/blob/master/bin/git-submodule-rewrite, but for the sake of proper stackoverflow answer protocol, I've included the solution in its entirety below.
Usage:
$ git-submodule-rewrite <submodule-name>
git-submodule-rewrite:
#!/usr/bin/env bash
# This script builds on the excellent work by Lucas Jenß, described in his blog
# post "Integrating a submodule into the parent repository", but automates the
# entire process and cleans up a few other corner cases.
# https://x3ro.de/2013/09/01/Integrating-a-submodule-into-the-parent-repository.html
function usage() {
echo "Merge a submodule into a repo, retaining file history."
echo "Usage: $0 <submodule-name>"
echo ""
echo "options:"
echo " -h, --help Print this message"
echo " -v, --verbose Display verbose output"
}
function abort {
echo "$(tput setaf 1)$1$(tput sgr0)"
exit 1
}
function request_confirmation {
read -p "$(tput setaf 4)$1 (y/n) $(tput sgr0)"
[ "$REPLY" == "y" ] || abort "Aborted!"
}
function warn() {
cat << EOF
This script will convert your "${sub}" git submodule into
a simple subdirectory in the parent repository while retaining all
contents and file history.
The script will:
* delete the ${sub} submodule configuration from .gitmodules and
.git/config and commit it.
* rewrite the entire history of the ${sub} submodule so that all
paths are prefixed by ${path}.
This ensures that git log will correctly follow the original file
history.
* merge the submodule into its parent repository and commit it.
NOTE: This script might completely garble your repository, so PLEASE apply
this only to a fresh clone of the repository where it does not matter if
the repo is destroyed. It would be wise to keep a backup clone of your
repository, so that you can reconstitute it if need be. You have been
warned. Use at your own risk.
EOF
request_confirmation "Do you want to proceed?"
}
function git_version_lte() {
OP_VERSION=$(printf "%03d%03d%03d%03d" $(echo "$1" | tr '.' '\n' | head -n 4))
GIT_VERSION=$(git version)
GIT_VERSION=$(printf "%03d%03d%03d%03d" $(echo "${GIT_VERSION#git version}" | tr '.' '\n' | head -n 4))
echo -e "${GIT_VERSION}\n${OP_VERSION}" | sort | head -n1
[ ${OP_VERSION} -le ${GIT_VERSION} ]
}
function main() {
warn
if [ "${verbose}" == "true" ]; then
set -x
fi
# Remove submodule and commit
git config -f .gitmodules --remove-section "submodule.${sub}"
if git config -f .git/config --get "submodule.${sub}.url"; then
git config -f .git/config --remove-section "submodule.${sub}"
fi
rm -rf "${path}"
git add -A .
git commit -m "Remove submodule ${sub}"
rm -rf ".git/modules/${sub}"
# Rewrite submodule history
local tmpdir="$(mktemp -d -t submodule-rewrite-XXXXXX)"
git clone "${url}" "${tmpdir}"
pushd "${tmpdir}"
local tab="$(printf '\t')"
local filter="git ls-files -s | sed \"s/${tab}/${tab}${path}\//\" | GIT_INDEX_FILE=\${GIT_INDEX_FILE}.new git update-index --index-info && mv \${GIT_INDEX_FILE}.new \${GIT_INDEX_FILE}"
git filter-branch --index-filter "${filter}" HEAD
popd
# Merge in rewritten submodule history
git remote add "${sub}" "${tmpdir}"
git fetch "${sub}"
if git_version_lte 2.8.4
then
# Previous to git 2.9.0 the parameter would yield an error
ALLOW_UNRELATED_HISTORIES=""
else
# From git 2.9.0 this parameter is required
ALLOW_UNRELATED_HISTORIES="--allow-unrelated-histories"
fi
git merge -s ours --no-commit ${ALLOW_UNRELATED_HISTORIES} "${sub}/master"
rm -rf tmpdir
# Add submodule content
git clone "${url}" "${path}"
rm -rf "${path}/.git"
git add "${path}"
git commit -m "Merge submodule contents for ${sub}"
git config -f .git/config --remove-section "remote.${sub}"
set +x
echo "$(tput setaf 2)Submodule merge complete. Push changes after review.$(tput sgr0)"
}
set -euo pipefail
declare verbose=false
while [ $# -gt 0 ]; do
case "$1" in
(-h|--help)
usage
exit 0
;;
(-v|--verbose)
verbose=true
;;
(*)
break
;;
esac
shift
done
declare sub="${1:-}"
if [ -z "${sub}" ]; then
>&2 echo "Error: No submodule specified"
usage
exit 1
fi
shift
if [ -n "${1:-}" ]; then
>&2 echo "Error: Unknown option: ${1:-}"
usage
exit 1
fi
if ! [ -d ".git" ]; then
>&2 echo "Error: No git repository found. Must be run from the root of a git repository"
usage
exit 1
fi
declare path="$(git config -f .gitmodules --get "submodule.${sub}.path")"
declare url="$(git config -f .gitmodules --get "submodule.${sub}.url")"
if [ -z "${path}" ]; then
>&2 echo "Error: Submodule not found: ${sub}"
usage
exit 1
fi
if ! [ -d "${path}" ]; then
>&2 echo "Error: Submodule path not found: ${path}"
usage
exit 1
fi
main
-
Does not work on Ubuntu 16.04. I sent a pull request to the Github repo.– qzncCommented May 3, 2017 at 11:09
-
2Good catch, @qznc. This was tested on OSX. I'll happily merge that when it passes on both platforms.– jsearsCommented May 3, 2017 at 15:09
-
-
Thank you so much for this! I got it to work perfectly in Windows 10 by grabbing your latest script from GitHub and running it under WSL.– SamCommented Sep 3, 2017 at 2:11
-
2Do all work without errors in Git Bash 2.20.1.1 on Windows 10 with latest version from github:
curl https://raw.githubusercontent.com/jeremysears/scripts/master/bin/git-submodule-rewrite > git-submodule-rewrite.sh
and./git-submodule-rewrite.sh <submodule-name>
– AlexeyCommented Dec 6, 2019 at 15:35
Since git 1.8.5 (Nov 2013) (without keeping the history of the submodule):
mv yoursubmodule yoursubmodule_tmp
git submodule deinit yourSubmodule
git rm yourSubmodule
mv yoursubmodule_tmp yoursubmodule
git add yoursubmodule
That will:
- unregister and unload (ie delete the content of) the submodule (
deinit
, hence themv
first), - clean up the
.gitmodules
for you (rm
), - and remove the special entry representing that submodule SHA1 in the index of the parent repo (
rm
).
Once the removal of the submodule is complete (deinit
and git rm
), you can rename the folder back to its original name and add it to the git repo as a regular folder.
Note: if the submodule was created by an old Git (< 1.8), you might need to remove the nested .git
folder within the submodule itself, as commented by Simon East
If you need to keep the history of the submodule, see jsears's answer, which uses git filter-branch
.
-
5This actually does delete it from the working tree in 1.8.4 (my entire submodule dir was cleared). Commented Oct 11, 2013 at 7:07
-
@ChrisDown you mean, the
deinit
alone cleaned the working tree from your submodule?– VonCCommented Oct 11, 2013 at 7:40 -
Yeah, it removes all the content in the submodule directory. Commented Oct 11, 2013 at 7:47
-
2@mschuett no, you are not missing anything: a submodule does not have a .git in it in the first place. If that was the case for you, it was a nested repo, not a submodule. That explains why this answer above would not apply in your case. For the difference between the two, see stackoverflow.com/a/34410102/6309.– VonCCommented Mar 13, 2016 at 20:14
-
1@VonC I'm currently on 2.9.0.windows.1, however the submodules may have been created several years ago on a much earlier version of git, I'm not sure. I think the steps seem to work as long as I remove that file before doing the final add + commit.– Simon E.Commented Jun 30, 2017 at 5:12
git rm --cached the_submodule_path
- remove the submodule section from the
.gitmodules
file, or if it's the only submodule, remove the file. - do a commit "removed submodule xyz"
git add the_submodule_path
- another commit "added codebase of xyz"
I didn't find any easier way yet. You can compress 3-5 into one step via git commit -a
- matter of taste.
-
6Shouldn't it be
.gitmodules
instead of.submodules
? Commented Feb 18, 2015 at 3:32 -
1
-
4I had to remove the
.git
directory of the submodule beforegit add
would work on the submodule folder– CarsonCommented Dec 12, 2018 at 15:48 -
Seconding Carson Evans, You definitely have to remove the .git file at the root of the submodule. This should be step 2.5. Commented Jul 14, 2020 at 12:48
Lots of answers here but all of them seem to be overly complex and likely do not do what you want. I am sure most people want to keep their history.
For this example the main repo will be [email protected]:main/main.git
and the submodule repo will be [email protected]:main/child.git
. This assumes that the submodule is located in the root directory of the parent repo. Adjust the instructions as needed.
Start by cloning the parent repo and removing the old submodule.
git clone [email protected]:main/main.git
git submodule deinit child
git rm child
git add --all
git commit -m "remove child submodule"
Now we will add the child repos upstream to the main repo.
git remote add upstream [email protected]:main/child.git
git fetch upstream
git checkout -b merge-prep upstream/master
The next step assumes that you want to move the files on the merge-prep branch into the same location as the submodule was above although you can easily change the location by changing the file path.
mkdir child
move all folders and files except the .git folder into the child folder.
git add --all
git commit -m "merge prep"
Now you can simply merge your files back into the master branch.
git checkout master
git merge merge-prep # --allow-unrelated-histories merge-prep flag may be required
Look around and make sure everything looks good before running git push
The one thing you have to remember now is that git log does not by default follow moved files however by running git log --follow filename
you can see the full history of your files.
-
2I got all the way to the final
git merge merge-prep
and received the errorfatal: refusing to merge unrelated histories
. Workaround is this:git merge --allow-unrelated-histories merge-prep
. Commented Dec 28, 2016 at 18:46 -
@humblehacker thanks i added a little comment in case others run into this as well. Commented Dec 29, 2016 at 0:35
-
1The best answer to keep history of submodule. Thank you @mschuett Commented Nov 7, 2017 at 10:48
-
In the example here, is there any way to fetch the upstream's files into the
child
directory, so you don't have to move them later? I have the same file names in a submodule and the main repo... so I just get a merge conflict since it's trying to merge the two files together.– SkittermCommented Jan 1, 2018 at 23:24 -
1@gianpaolo if you rebase/merge master into them then yes. Commented Apr 18, 2023 at 22:33
It happened to us that we created 2 repositories for 2 projects that were so coupled that didn't make any sense to have them separated, so we merged them.
I'll show how to merge the master branches in each first and then I will explain how you can extend this to every branches you got, hope it helps you.
If you got the submodule working, and you want to convert it to a directory in place you can do:
git clone project_uri project_name
Here we do a clean clone to work. For this process you don't need to initialize or update the submodules, so just skip it.
cd project_name
vim .gitmodules
Edit .gitmodules
with your favorite editor (or Vim) to remove the submodule you plan to replace. The lines you need to remove should look something like this:
[submodule "lib/asi-http-request"]
path = lib/asi-http-request
url = https://github.com/pokeb/asi-http-request.git
After saving the file,
git rm --cached directory_of_submodule
git commit -am "Removed submodule_name as submodule"
rm -rf directory_of_submodule
Here we remove the submodule relation completely so we can create bring the other repo to the project in-place.
git remote add -f submodule_origin submodule_uri
git fetch submodel_origin/master
Here we fetch the submodule repository to merge.
git merge -s ours --no-commit submodule_origin/master
Here we start a merge operation of the 2 repositories, but stop before commit.
git read-tree --prefix=directory_of_submodule/ -u submodule_origin/master
Here we send the content of master in the submodule to the directory where it was before prefixing a directory name
git commit -am "submodule_name is now part of main project"
Here we complete the procedure doing a commit of the changes in the merge.
After finishing this you can push, and start again with any other branch to merge, just checkout the branch in you repository that will receive the changes and change the branch you bringing in the merge and read-tree operations.
-
this doesn't seemed to have preserved history of the submodule files, I just see a single commit in the git log for the files added under
directory_of_submodule
Commented Jan 21, 2013 at 12:47 -
@Anentropic Sorry for the delay to reply. I just did the full procedure again (with a small fix). The procedure keeps the whole history, but it has a merge point, maybe that is why you don't find it. If you want to see the submodule history just do a "git log", lookup for the merge commit (in the example is the one with message "submodule_name is now part of main project"). It will have 2 parent commits (Merge: sdasda asdasd), git log the second commit and you got all your submodule/master history there.– dvicinoCommented Apr 24, 2013 at 9:31
-
my memory is hazy now but I think I was able to get the history of the merged submodule files by doing
git log original_path_of_file_in_submodule
i.e. the path registered in the git repo for the file (which doesn't any longer exist on the filesystem) even though the submodule file now lives atsubmodule_path/new_path_of_file
Commented Apr 24, 2013 at 10:00 -
This doesn't preserve the history very well, and also the paths are wrong. I feel that something like a tree-filter is needed but I'm out of my depth... trying what I've found here: x3ro.de/2013/09/01/…– Luke HCommented Aug 29, 2014 at 12:46
-
This answer is obsolete, stackoverflow.com/a/16162228/11343 (VonC answer) does the same but better– CharlesBCommented Jan 8, 2018 at 11:28
The best answer to this I have found is here:
http://x3ro.de/2013/09/01/Integrating-a-submodule-into-the-parent-repository.html
This article explains the procedure very well.
Here's a slightly improved version (IMHO) of @gyim's answer. He is doing a bunch of dangerous changes in the main working copy, where I think it's much easier to operate on separate clones and then merge them together at the end.
In a separate directory (to make mistakes easier to clean up and try again) check out both the top repo and the subrepo.
git clone ../main_repo main.tmp
git clone ../main_repo/sub_repo sub.tmp
First edit the subrepo to move all files into the desired subdirectory
cd sub.tmp
mkdir sub_repo_path
git mv `ls | grep -v sub_repo_path` sub_repo_path/
git commit -m "Moved entire subrepo into sub_repo_path"
Make a note of the HEAD
SUBREPO_HEAD=`git reflog | awk '{ print $1; exit; }'`
Now remove the subrepo from the main repo
cd ../main.tmp
rmdir sub_repo_path
vi .gitmodules # remove config for submodule
git add -A
git commit -m "Removed submodule sub_repo_path in preparation for merge"
And finally, just merge them
git fetch ../sub.tmp
# remove --allow-unrelated-histories if using git older than 2.9.0
git merge --allow-unrelated-histories $SUBREPO_HEAD
And done! Safely and without any magic.
-
... which answer is that? Might want to reference the username as well as the top answer can change over time.– ContangoCommented Oct 22, 2019 at 10:05
-
@Contango answer updated. but the top answer is still the top answer by a 400 point lead ;-)– datalessCommented Oct 23, 2019 at 19:08
-
Does this work if the subrepo already contains a directory named
subrepo
with stuff in it?– detlyCommented Dec 17, 2019 at 23:04 -
In the last step I get following error:
git merge $SUBREPO_HEAD fatal: refusing to merge unrelated histories
Should I usegit merge $SUBREPO_HEAD --allow-unrelated-histories
in this case? Or should it work without and I made a mistake?– Ti-mCommented Jan 15, 2020 at 10:54 -
1@Ti-m Yes, this is exactly the case of merging two histories that do not share any commits. The guard against unrelated histories seems to be new in git since I first wrote this; I'll update my answer.– datalessCommented Jan 16, 2020 at 17:06
For when
git rm [-r] --cached submodule_path
returns
fatal: pathspec 'emr/normalizers/' did not match any files
Context: I did rm -r .git*
in my submodule folders before realizing that they needed to be de-submoduled in the main project to which I had just added them. I got the above error when de-submoduling some, but not all of them. Anyway, I fixed them by running, (after, of course, the rm -r .git*
)
mv submodule_path submodule_path.temp
git add -A .
git commit -m "De-submodulization phase 1/2"
mv submodule_path.temp submodule_path
git add -A .
git commit -m "De-submodulization phase 2/2"
Note that this doesn't preserve history.
Based on VonC's answer, I have created a simple bash script that does this. The add
at the end has to use wildcards otherwise it will undo the previous rm
for the submodule itself. It's important to add the contents of the submodule directory, and not to name the directory itself in the add
command.
In a file called git-integrate-submodule
:
#!/usr/bin/env bash
mv "$1" "${1}_"
git submodule deinit "$1"
git rm "$1"
mv "${1}_" "$1"
git add "$1/**"
In main repo
- git rm --cached [submodules_repo]
- git commit -m "Submodules removed."
- git push origin [master]
In submodules repo
- rm -rf .git
Again main repo
- git add [submodules_repo]
- git add .
- git commit -m "Submodules repo added into main."
- git push origin [master]
-
There are already several answers here, and most of them include lots of explanation. Does this answer introduce new information? Please read How to Answer.– ChrisCommented Nov 25, 2021 at 20:31
-
I found it more convenient to (also?) fetch local commit data from the submodule, because otherwise I would loose them. (Could not push them as I have not access to that remote). So I added submodule/.git as remote_origin2, fetched it commits and merged from that branch. Not sure if I still need the submodule remote as origin, since I am not familiar enough with git yet.
Here's what I found best & simplest.
In submodule repo, from HEAD you want to merge into main repo:
git checkout -b "mergeMe"
mkdir "foo/bar/myLib/"
(identical path as where you want the files on main repo)git mv * "foo/bar/myLib/"
(move all into path)git commit -m "ready to merge into main"
Back in main repo after removing the submodule and clearing the path "foo/bar/myLib":
git merge --allow-unrelated-histories SubmoduleOriginRemote/mergeMe
boom done
histories preserved
no worries
Note this nearly identical to some other answers. But this assumes you own submodule repo. Also this makes it easy to get future upstream changes for submodule.
git submodule deinit
, see my answer belowgit submodule deinit asubmodule ; git rm asubmodule
is enough, as illustrated in my answer below