git p4: migrate full history – including integration history

A couple of years ago, a subfolder in one of my Perforce depots (e.g. //FirstDepot/LargeSubfolder/...) became too large to maintain, so I ‘migrated’ it into a depot of its own (e.g. //NewDepotFromLargeSubfolder/...). Perforce does not support integrations from one depot to another, so I basically checked out the files, added them to the new depot and deleted them from the subfolder.

Now I am about to migrate the entire Perforce server to a number of git repositories. The git p4 procedure works very well, I already migrated other, unrelated Perforce depots to git successfully (including their full file history using @all – but then, those depots were never ‘migrated’ within Perforce in the way described…).

  • Docker Jenkins & Sonar setup
  • git imap-send with Gmail
  • Gitlab not sending email when accessed by https
  • Git filter-branch commit-filter debugging output
  • Heroku releases - how to roll forward after rolling back?
  • git conflict - file deleted in HEAD and modified in revision
  • The problem is that I do not want to lose the file history underneath //FirstDepot/LargeSubfolder/.... There are a couple thousand changelists in there (plus a couple thousand more at //NewDepotFromLargeSubfolder/...) and many of them contain very valuable information.

    I tried to migrate the file history from the first location and then ‘stack’ the history of the second location on top of it:

    host:~ user$ mkdir SomeTempFolder
    host:~ user$ cd SomeTempFolder
    host:SomeTempFolder user$ export P4PORT=perforce:1666
    host:SomeTempFolder user$ git p4 clone //FirstDepot/LargeSubfolder@all gitRepoFolder
    Importing from //FirstDepot/LargeSubfolder@all into gitRepoFolder
    Initialized empty Git repository in /Users/user/SomeTempFolder/gitRepoFolder/.git/
    Import destination: refs/remotes/p4/master
    Importing revision 26776 (99%)
    Ignoring apple filetype file //FirstDepot/LargeSubfolder/SomePath/SomeIconFile.icns
    Importing revision 26778 (100%)
    

    So apart from a message about a Mac OS X icon file being ignored, the first bit works. The second part fails however:

    host:SomeTempFolder user$ git p4 clone //NewDepotFromLargeSubfolder@all gitRepoFolder
    Importing from //NewDepotFromLargeSubfolder@all into gitRepoFolder
    Reinitialized existing Git repository in /Users/user/SomeTempFolder/gitRepoFolder/.git/
    Import destination: refs/remotes/p4/master
    Importing revision 26777 (0%)
    Ignoring apple filetype file //NewDepotFromLargeSubfolder/SomePath/SomeIconFile.icns
    Importing revision 27142 (33%)
    Ignoring apple filetype file //NewDepotFromLargeSubfolder/AnotherPath/Some AppleScript file.scpt
    Ignoring apple filetype file //NewDepotFromLargeSubfolder/AnotherPath/Another AppleScript file.scpt
    Importing revision 27620 (100%)
    
    fast-import failed: warning: Not updating refs/remotes/p4/master (new tip <some long git ID> does not contain <another long git ID>)
    git-fast-import statistics:
    <loads of git statistic>
    

    Running git p4 with --verbose seems to indicate that the actual importing of Perforce revisions completes successfully, but then something goes wrong which I don’t understand:

    Importing revision 27620 (100%)commit into refs/remotes/p4/master
    Opening pipe: ['p4', '-G', '-x', '-', 'print']
    Path/of/the/file/in/the/last/Changelist
    
    Reading pipe: ['git', 'config', '--bool', 'git-p4.importLabels']
    Traceback (most recent call last):
      File "/usr/libexec/git-core/git-p4", line 3287, in <module>
        main()
      File "/usr/libexec/git-core/git-p4", line 3281, in main
        if not cmd.run(args):
      File "/usr/libexec/git-core/git-p4", line 3155, in run
        if not P4Sync.run(self, depotPaths):
      File "/usr/libexec/git-core/git-p4", line 3030, in run
        die("fast-import failed: %s" % self.gitError.read())
      File "/usr/libexec/git-core/git-p4", line 106, in die
        raise Exception(msg)
    
    Exception: fast-import failed: warning: ... (as above) ...
    

    The migration of the second part works fine when done into a new git repository.

    After reading `git-p4 clone` fails "new tip … does not contain …", I retried the migration in a working folder without spaces; same result. I’m working on Mac OS X btw, but the Perforce server is running on a Debian 7.5 virtual machine.

    My questions:
    + Does anyone have some advice how to find out what’s wrong here ?
    + Is it generally possible to ‘stack’ chunks of revision history ?
    + Can anyone think of an alternate approach (apart from having two git repos) ?

    I’m not a git expert at all, but so far, I could pretty much just throw stuff at it and git did an excellent job of figuring out what I want to do; also the Reinitialized existing Git repository message seems to indicate that I’m not totally off.

  • How does one disable Git's HTML help (on Windows)?
  • Git Switching branch
  • How to define an alternative remote url for a git remote repository?
  • How to remove cloned git repo from a second git repo?
  • Cannot add files, not a git repository
  • Git adding files to repo
  • One Solution collect form web for “git p4: migrate full history – including integration history”

    The migration of the second part works fine when done into a new git repository

    That could be the basis of an acceptable workaround here: have to separate Git repo, and chose a technique from “How do you merge two git repositories?” to get one final repo.

    The git filter-branch approach is detailed in “Combining multiple git repositories”.

    (You also have a simpler approach based on merge, also described here: The OP ssc selected “Merge two Git repositories without breaking file history”)

    Git Baby is a git and github fan, let's start git clone.