Git local mirror and repository
Here is what I would like to do:
- Have a local git repository that mirrors an upstream one
- Be able to push “local” branches / changes to that repository and keep those locally
- Keep this repository in sync with the upstream one, including:
- Fetch any new branch
- Delete any reference of branches that are deleted upstream
I setup my cron job to fetch all the changes from upstream and prune any branch that have been deleted like this:
- Mirroring git repository with submodules
- Mirroring from Gitlab to Github
- How can I keep a git mirror in sync (including deleting branches)?
- how to sync a repo mirror to a new upstream repo
- Gitlab returning remote: You do not have permissions to do this. while performing scripted git push
- How to update a git clone --mirror?
*/5 * * * * cd /home/git/myrepo.git && git fetch origin && git remote prune origin > /dev/null
So far what I have tried (and why it failed):
1- Setup the git repository as a mirror (as described here)
git clone --bare --mirror URL
The problem with that is when it does the
git remote prune, it is also deleting references to the “local” changes that have been pushed there (and not to the upstream server).
I also tried to have this local repository be the mirror for two separate repositories (with the same master but some different branches) and I hit a similar problem when doing
git remote prune, it will delete the branches coming from the other repository.
2- Setup git only as a bare repository:
git clone --bare URL
git fetch origin is not updating properly, it seems to be downloading the objects, but does not create the refs and then only prints
* branch HEAD -> FETCH_HEAD
and the “location” of the current branches is not being updated with what’s in the upstream server.
I also tried
git remote update as described here, with the same result.
I can convert that repository as a mirror with:
git config remote.origin.fetch 'refs/heads/*:refs/heads/*'
But that only brings me back to the problem in (1)
One Solution collect form web for “Git local mirror and repository”
Assuming you can drop the “mirror” requirement, and have “local (bare) repo $X also copies upstream repo $UX using refs/heads/upstream/$branch to name upstream branches known there as refs/heads/$X”, use your second approach, but do this instead:
$ cd /tmp; mkdir tt; cd tt; git clone --bare ssh://$upstream_host/tmp/t $ cd t.git $ git config remote.origin.fetch '+refs/heads/*:refs/heads/upstream/*' $ git fetch -p # accidentally omitted this step from cut/paste earlier
This assumes you won’t use branch names like
upstream/master for anything yourself. (You could also/instead do something like:
git config remote.origin.fetch '+refs/*:refs/upstream/*'
refs/upstream/* references are not copied over by normal
git fetch, etc., so this is more of a pain for “normal” git users.)
Let’s make a clone of the
--bare repo too to see what happens as we go on. (For reference, on
$upstream_host, I have
/tmp/t, a regular git repo. On
$local_host, the not-quite-mirror machine, I have
--bare repo that does this upstream tracking thing. I am actually using the same host for both but the principle applies…)
$ cd /tmp; mkdir xt; cd xt; git clone ssh://$local_host/tmp/tt/t.git Cloning into 't'... remote: Counting objects: 96, done. remote: Compressing objects: 100% (54/54), done. remote: Total 96 (delta 33), reused 96 (delta 33) Receiving objects: 100% (96/96), 17.11 KiB | 0 bytes/s, done. Resolving deltas: 100% (33/33), done. Checking connectivity... done
Now I made a change on
/tmp/t, and commited it. Back on
$ cd /tmp/tt/t.git; git fetch -p origin # -p will prune deleted upstream/foo's remote: Counting objects: 5, done. remote: Compressing objects: 100% (4/4), done. remote: Total 4 (delta 1), reused 0 (delta 0) Unpacking objects: 100% (4/4), done. From ssh://$host/tmp/t + c10e54c...5e01371 master -> upstream/master (forced update)
Thus, changes made upstream will appear in your “sort of a mirror but not exactly” bare git repo as a change to
upstream/master rather than
master, or more generally,
upstream/$branch for any
$branch. If you want to merge them you’ll have to do that manually. My example below is a bit messy because the change I made on
$upstream_host was a history rewrite (hence all the
forced update stuff), which winds up getting exposed here via the clones. If you don’t want it exposed you’ll have to note which updates were history rewrites and (in effect) manually copy them to your own not-quite-mirror, and then on to any clones of that. I’ll just go ahead and make a real merge.
So, now we go to the non-bare repo on
$ cd /tmp/xt/t $ git fetch remote: Counting objects: 5, done. remote: Compressing objects: 100% (4/4), done. remote: Total 4 (delta 1), reused 1 (delta 0) Unpacking objects: 100% (4/4), done. From ssh://$local_host/tmp/tt/t + c10e54c...5e01371 upstream/master -> origin/upstream/master (forced update) $ git status # On branch master nothing to commit, working directory clean $ git log --oneline --decorate --graph * 5e01371 (origin/upstream/master) add ast example | * c10e54c (HEAD, origin/master, origin/HEAD, master) add ast example |/ * 309b36c add like_min.py ... [snipped] $ git merge origin/upstream/master Merge remote-tracking branch 'origin/upstream/master' # Please enter a commit message to explain why this merge is necessary, # especially if it merges an updated upstream into a topic branch. # # Lines starting with '#' will be ignored, and an empty message aborts # the commit. ... $ git push warning: push.default is unset; its implicit value is changing in Git 2.0 from 'matching' to 'simple'. To squelch this message ... Counting objects: 1, done. Writing objects: 100% (1/1), 244 bytes | 0 bytes/s, done. Total 1 (delta 0), reused 0 (delta 0) To ssh://$local_host/tmp/tt/t.git c10e54c..e571182 master -> master
I’ve now updated the
--bare clone (
/tmp/tt/t.git) via the non-bare clone to merge the upstream work into my local not-exactly-a-mirror. The
HEAD revision is my merge,
HEAD^1 is the original (broken) update that used to be
origin/upstream/master (before all the “forced update”ing), and
HEAD^2 is the corrected update that is now
$ git rev-parse HEAD^2 origin/upstream/master 5e013711f5d6eb3f643ef562d49a131852aa4aa1 5e013711f5d6eb3f643ef562d49a131852aa4aa1
(The name is just
upstream/master in the
--bare clone, so the
git rev-parse above is from