My project uses over 100 git submodules, which submodule alternative can handle a lot of repositories gracefully

I’ve been researching git subtree and other alternatives to git submodules. My project has well over 100 submodules and it’s very unwieldy to manage them all.

Can anyone recommend a workflow that works really well with a large number of repositories that need to be kept in sync.

  • Resolve recursive git cookbook dependencies with Berkshelf
  • Link to base source code in GIT (C#)
  • Pulling my own private repository from Bitbucket via Composer
  • How to make dependencies between merged patch and before merged patch in gerrit?
  • Jar Dependencies in GitHub
  • How to organize git in projects with overlapping dependencies?
  • using git post-receive for overwriting files in multiple directories
  • Git pull keeps deleting everything I've added
  • Track branch after creation?
  • git status - list last modified date
  • How to solve merge conflicts if I only want to pull files to local server?
  • hudson.util.HudsonFailedToLoad error in Jenkins
  • One Solution collect form web for “My project uses over 100 git submodules, which submodule alternative can handle a lot of repositories gracefully”

    If you project has over 100 git submodules of components and dependencies, their management will be unwieldy no matter which approach you use 🙂 I suggest look for ways to script and automate as many parts as possible. Trust me, the novelty of playing with and chaining git commands wear out very quickly for most people, especially when deadlines are approaching. There is already a very good answer here on the comparison of the different approaches to manage git sub-projects.

    Regarding workflow, I will first separate repositories that are under your control from those that aren’t i.e. 3rd party repositories.

    For 3rd party repositories which don’t change often (either via merges or upstream PRs), you can still use submodules. Typically, you will point these submodules to the HEAD of some stable tags. Sync-ing them it’s just a matter of running (or scripting) git submodule update --recursive --remote. If these 3rd party dependencies can be specified in package management tools like bundler (for ruby projects), it will help to simplify your subprojects management.

    For repositories that your own and change often, either gitslave or git-subtree are two alternatives, depending on your team’s preferences.

    gitslave multiplexes git operations into multiple branches. IOW, when you branch, merge, commit, push, pull etc., each command will be run on the parent project and all slaves in turn. This mandates the team to work in a top-down manner, starting from the super-project down to the slaves.

    gitsubtree uses Git’s subtree merge functionality to achieve a similar effect as submodules, by actually storing the files in the main repository and merging in changes directly to that repository. The end result is a canonical repository with the option of including all the subprojects’ history. In a way, this allows team members to focus more on the subtrees they are responsible for, but will require extra work to merge back to the parent tree.

    As a developer, my preference is to work at the lower sub-projects level (to do my “red, green, refactor” cycle), and touch the parent projects only when necessary. But regardless of whether you choose a top-down or bottom-up workflow, try to identify repetitive error-prone steps in your branching & merging strategy, and script them as much as possible.

    Git Baby is a git and github fan, let's start git clone.