Is there an effort to develop build-oriented file system with automatic change detection of files?

I recently started to use Git.
One of the interesting features I discovered was the use of hashes to quickly detect changes.

On the other hand, I see that build tools (like make, ant, javac, etc.) tries to detect changes in source files by checking file’s timestamp.

  • NuGet Enterprise - best practices for different maturity levels of packages
  • git repository gives 403 error while connecting from jenkins in linux
  • Builds section not showing in Visual Studio 2013 for GIT projects
  • Error building iOS app on Jenkins (MacOSX server)
  • Configure a Jenkins job to select an application and trunk/tags/branches from SVN
  • Problems getting my ANT builds to work after OS upgrade
  • The problems in this approach are:

    1. If you work on more than one
      machine, you have to make sure all
      clocks are in sync, otherwise, a
      new file may be considered
      unchanged because the other machine’s clock gave it timestamp of the past relative to the build machine.
    2. On a big project, you have to scan all files’ timestamp in order to detect a change.

    I wonder if someone has already taken the Git approach in order to deal with these issues:

    1. Each file has a unique hash, depending on its content, not timestamp.
    2. Each directory also has its hash, depending on the files in the directory and their hashes.
    3. Even the simple change deeply inside the source tree causes the root directory to have a different hash due to the above rules

    Such a mechanism could help making build tools much faster, because detecting a change in source tree is a simple operation of hash comparison. If the hash of source tree root directory has changed, it means that a change occurred deeper in the source tree, so continue to scan the tree recursively for changes – exactly as Git does to detect changes.

    It doesn’t necessarily mean that this source tree has to be managed by Git.
    My idea is that the file system would automatically provide file’s hash code as one of its attributes / metadata, so the build tool could rely on this instead of on timestamp. And in addition, directory hash would automatically reflect the state of the file in it.

    I already read a little bit about Sun’s ZFS, but I am not sure it’s a complete solution to make builds faster.

    What do you think about this idea?
    Is there already such file system?
    Is there already such build tool?

  • Building akka fails on “Error: could not retrieve JNA”
  • Dynamic version from git with go get
  • Tracking code versions in an executable
  • git status shows modified after files copied
  • Jenkins marks good build as failure because of test failed
  • How to get the last push date or file modification date when deploying to Heroku?
  • One Solution collect form web for “Is there an effort to develop build-oriented file system with automatic change detection of files?”

    I’ll argue that what you’re trying to solve is actually a non-issue:

    The clock skew problem can be mostly avoided by using NTP.

    Certainly it’d be nice to have clock skew issues eliminated entirely, but we can probably agree that throwing a fairly complex content-tracking system at the problem is overkill.

    Regarding performance, scanning the entire tree tends to not be a problem in practice. stat is ridiculously fast (so long as you’re not on Windows) — ls -lR > /dev/null over the entire Linux kernel tree (38k files) takes 350 ms on my system.

    In fact, if stat’ing all your files is a problem, then your version control system will become slow, and that will be a much bigger problem than your build performance. Every git status or git diff, for instance, stats all files in your working copy to check their mtimes, so you’d better hope that’s fast.

    So if you’re looking to speed up make, don’t look at the file system; it’s most likely insignificant compared to whatever is actually eating up your build time.

    Hope that eases your mind!

    Git Baby is a git and github fan, let's start git clone.