3

I'm attempting to create a node/go library (https://github.com/simpleviewinc/git-tools) to help using git to checkout a repo to a specific remote/branch for handling production checkouts as well as developer checkouts for peer review. In both cases when a developer executes the checkout command I want it to execute and setup the working copy to the exact state declared in the remote repositories. So this means it will toss any untracked changes, unpushed commits, all of that. It is to make their local working copy or the server working copy exactly match the remotes. In production this is obviously so that the live server matches the exact state. On local, it ensures that the developer has the exact state of the fork that they are reviewing (with no changes on their local box interfering with the review). On local, it will prompt the user before doing destructive actions (like git reset, git clean).

The problem I am facing is that I cannot figure out a set of git commands that will consistently work in all cases when switching between branches if the branch has submodules. In my test, I have one repo with 3 branches, one branch has no submodules (master), another module has 1 submodule (submodule-test), and another branch has another submodule (submodule-test2) that points to a different repository (at the same path). I want my library to be able to switch the working copy from any branch to any other branch, executing the exact same set of commands, without the developer executing needing to know the specific setup related to the destination branch. Basically, it should be "give me this code whatever it is declared in the remote". In example if a dev is peer reviewing and both the main repo and the submodule have forks/branches. I want the dev to be able to just gun git-tools checkout proj1 --remote=dev1 --branch=pr-150 and it will checkout dev1's fork of proj1 and branch pr-150. Then if they ran git-tools checkout proj1 it would switch it back to master of proj1.

Right now the closest set of commands for switching branches I've been able to almost get working is:

git submodule deinit --all
git checkout branch
git submodule sync
git submodule update

That almost works, except for it fails when switching from one branch with submodules to another branch with different submodules the first time (or a different remote for the same sub-module, such as a developer fork).

In example here is one set of commands and their failure in git 2.20.1

cd /tmp
git clone git@github.com:simpleviewinc/git-tools-test.git ./checkout --recurse-submodules
cd checkout
git checkout submodule-test
git submodule sync
git submodule update
# branch submodule-test fully checked out, all submodules downloaded, looking good!
git submodule deinit --all
git checkout submodule-test2
git submodule sync
git submodule update
fatal: remote error: upload-pack: not our ref c1bba6e3969937125248ee46e308a8efec8ac654
Fetched in submodule path 'submodule', but it did not contain c1bba6e3969937125248ee46e308a8efec8ac654. Direct fetching of that commit failed.

It fails because it uses the wrong submodule remote, even though I thought that was the explicit purpose of submodule sync. If I go from submodule-test to master, it will succeed, but if master has a submodule it fails, so that doesn't help.

I tried --recurse-submodules but that fails too, but this time when checking from a branch without submodules to a branch with submodules.

cd /tmp
git clone git@github.com:simpleviewinc/git-tools-test.git ./checkout --recurse-submodules
cd checkout
git checkout submodule-test --recurse-submodules
fatal: not a git repository: ../.git/modules/submodule
fatal: could not reset submodule index

Master doesn't have submodules so when I switch to another branch, something goes awry.

There has to be some git incantation that will reliably allow you to switch from branch A to branch B that can be run for any branch A and any branch B regardless of the submodules in play. If you examine the repo I'm testing with, it's basically empty, so it's perfectly safe for you to execute the exact same commands and see the exact same errors I'm hitting. Ultimately it seems like something needs to be synchronized between .git/config, .git/modules/module-name/config and the .gitmodules, but I cannot figure out a rely set of sequences that meets the goal. Any assistance would be incredible as I've already spent way to long trying to beat my head against these stupid submodules.

1
  • 1
    I am not sure there is a way to do this. What would be nice for fixing the problem in a future Git release is to have some test repository build procedure, so that we could add a test, fix things, and make sure they stay fixed. (Well, test repositories, plural, here: to test this will obviously require at least 3 repositories.)
    – torek
    Nov 2, 2020 at 10:39

3 Answers 3

2

I was able to figure this out. It's not trivial but so far it's working. The key element is that prior to switching from one branch to another I stash the .git/modules folder somewhere based on the name of the current branch. This way when I switch back to that branch, I can restore the stashed modules since that stores the git repository information for all of the active submodules on that branch.

Roughly the process of going from any branch to any other branch is as follows:

export TARGET_BRANCH="my-branch-name"
export CURRENT_BRANCH=$(git rev-parse --abbrev-ref HEAD)
if [ -f ".gitmodules" ]; then
  git submodules deinit --all
  mkdir -p .git/git-tools/modules
  mv .git/modules .git/git-tools/modules/$CURRENT_BRANCH
fi

git checkout $TARGET_BRANCH

if [ -f ".gitmodules" ]; then
  if [ -f ".git/git-tools/modules/$TARGET_BRANCH" ]; then
    git mv .git/git-tools/modules/$TARGET_BRANCH .git/modules
  fi

  git submodule sync && git submodule update --init
fi
1

This question was asked on the Git mailing list and I replied there. Copying my answer, slightly edited, here:

Currently, the only method I've seen that you can reliably use to switch between different branches when they don't all have the same contained submodules comes from the Stack Overflow answer at https://stackoverflow.com/a/64690495/3319611. I'll reproduce the Bash snippet it presents as a solution here for completeness's sake:

export TARGET_BRANCH="my-branch-name"
export CURRENT_BRANCH=$(git rev-parse --abbrev-ref HEAD)
if [ -f ".gitmodules" ]; then
   git submodules deinit --all
   mkdir -p .git/git-tools/modules
   mv .git/modules .git/git-tools/modules/$CURRENT_BRANCH
fi

git checkout $TARGET_BRANCH

if [ -f ".gitmodules" ]; then
   if [ -f ".git/git-tools/modules/$TARGET_BRANCH" ]; then
     git mv .git/git-tools/modules/$TARGET_BRANCH .git/modules
   fi

   git submodule sync && git submodule update --init
fi

The script above is in my opinion complicated and uneeded. The bottom line is:

  1. git checkout --recurse-submodules $ref should always work in an ideal world, but there are still some missing pieces as of today (Git 2.33.0, September 2021).
  2. git checkout $ref && git submodule sync --recursive && git submodule update --recursive should always work.

Here goes:

The problem I am facing is that I cannot figure out a set of git commands that will consistently work in all cases when switching between branches if the branch has submodules. In my test, I have one repo with 3 branches, one branch has no submodules (master), another module [sic, should be "branch"] has 1 submodule (submodule-test), and another branch has another submodule (submodule-test2) that points to a different repository (at the same path).

[...]

In example here is one set of commands and their failure in git 2.20.1

cd /tmp
git clone git@github.com:simpleviewinc/git-tools-test.git ./checkout --recurse-submodules
cd checkout
git checkout submodule-test
git submodule sync
git submodule update
# branch submodule-test fully checked out, all submodules downloaded, looking good!
git submodule deinit --all
git checkout submodule-test2
git submodule sync
git submodule update
fatal: remote error: upload-pack: not our ref c1bba6e3969937125248ee46e308a8efec8ac654
Fetched in submodule path 'submodule', but it did not contain c1bba6e3969937125248ee46e308a8efec8ac654. Direct fetching of that commit failed.

It fails because it uses the wrong submodule remote, even though I thought that was the explicit purpose of submodule sync.

The reason it fails is because the git submodule sync did not update the remote.origin.url configuration in the Git config file of the submodule, i.e. /tmp/checkout/.git/modules/submodule/config, despite (confusingly) outputing "Synchronizing submodule url for 'submodule'". It does change the submodule.$name.url value in the config file of the superproject, though, (/tmp/checkout/.git/config) but git submodule update only uses remote.origin.url in the submodule's config file (if it exists).

The reason why the second git submodule sync does not change remote.origin.url is because the submodule has been deinitialized when the command is run, because the previous command is git submodule deinit --all, which deinitializes all submodules. So this second step of changing remote.origin.url is skipped for deinitialized submodules (this fact is missing from the doc).

The following sequence would be the correct one:

git clone git@github.com:simpleviewinc/git-tools-test.git ./checkout [--recurse-submodules] # --recurse-submodules is optional
cd checkout
git checkout submodule-test
git submodule update --init
git checkout submodule-test2
git submodule sync
git submodule update
# and we can switch back
git checkout submodule-test
git submodule sync
git submodule update

Note that the correct command to initialize all submodules is git submodule init or, to initialize, clone and check them out in a single step, git submodule update --init. The fact that the first git submodule sync works to initialize the submodule is in fact due to using --recurse-submodules for git clone, which sets submodule.active to the match-all pathspec . in the superproject's config, and git submodule sync recurses into active submodules (this is also missing from the doc).

So for git checkout, what you want to achieve can be done with a post-checkout hook:

#!/bin/sh

# If the checkout was a branch checkout [1], update the submodules
# to the commits recorded in the superproject
# [1] https://git-scm.com/docs/githooks#_post_checkout

previous_head=$1
new_head=$2
checkout_type_flag=$3

if [ "$checkout_type_flag" -eq 1 ] ; then
   git submodule sync --recursive
   git submodule update --init --recursive
fi

Here I add --recursive to both commands for extra safety in case any of your submodules themselves contain submodules.


Now, about git checkout --recurse-submodules:

I tried --recurse-submodules but that fails too, but this time when checking from a branch without submodules to a branch with submodules.

cd /tmp
git clone git@github.com:simpleviewinc/git-tools-test.git ./checkout --recurse-submodules
cd checkout
git checkout submodule-test --recurse-submodules
fatal: not a git repository: ../.git/modules/submodule
fatal: could not reset submodule index

Yeah, that one is bad. The reason it fails is because:

  1. git clone --recurse-submodules in fact pretty much runs a simple git submodule update --init --recursive at the end of the process, at the step where it's checking out the working tree. This means that only submodules present in the branch being checked out get initialized and cloned.

  2. git clone --recurse-submodules always writes submodule.active=. to the superproject's config.

  3. git checkout --recurse-submodules recurses into active submodules, and for that it needs access to the Git repository of the submodule, which does not exist yet since it was not cloned, and so it errors.

This is the same error you get when you try to recursively checkout an older revision that contains a submodule that was since deleted [1], [2].

I suggested in those threads a few ways all this could be improved; here is my up to date take on the subject:

  1. git clone --recurse-submodules should be able to at least clone all submodules for all branches that are cloned, and put their Git directory in .git/modules/. This would allow your use case to "just work" with git checkout --recurse-submodules.

  2. git clone --recurse-submodules could also be taught to clone all submodules for all revisions of all branches that are cloned. This would allow the "deleted submodule" cases I mentioned to work, but would not be wanted in all situations, so it could be a supplementary flag to git clone.

  3. git fetch should be taught to clone new submodules.

  4. git checkout --recurse-submodules could be taught to clone missing submodules and fetch missing submodules commits. This would cover both your use case as well as the "deleted submodule" use case.

  5. In any case, git checkout --recurse-submodules should not abort midway if it can't find the Git repository of the submodule and leave .gitmodules untracked in the working tree, and a config file in .git/modules/$name/ with only core.worktree set.

0

git checkout --recurse-submodules $commit doesn't work for me in cases of submodule url/path changes between current status and the $commit, I tried below commands which works well (which is also referenced here):

  git checkout $commit
  git submodule sync --recursive
  git submodule update --init --recursive
  git clean -ffdx

Here the key is submodule sync to refresh the submodule path/urls

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.