CMake ExternalProject: how to specify relative path to the root CMakeLists.txt?

It seems that CMake ExternalProject always assumes the root directory
of the external project to be the source directory. But what if that is not
the case?

Consider the following example:

  • TFS 2015: How to exclude certain folders from triggering build with Git
  • GitCloneTask doesn't clone repository (“The remote end hung up unexpectedly”)
  • Building akka fails on “Error: could not retrieve JNA”
  • How to properly use CI scripts with Git hooks to compress source
  • TFS 2015 Git Repository Build Get only Relevant files
  • How to mark a build unstable in Jenkins when running shell scripts
  • The external project uses this directory layout:

    libfoo.git                 <--- ExternalProject assumes this as source dir.
    ├── ...
    └── libfoo                 <--- However, the actual source directory is this!
        ├── CMakeLists.txt
        └──  ...
    

    In the depending project libfoo is configured like this:

    ExternalProject_Add( libfoo 
        PREFIX            "${CMAKE_CURRENT_BINARY_DIR}/EP_libfoo"
        GIT_REPOSITORY    "<link to remote which hosts libfoo.git>" 
        GIT_TAG           "<some hash>"
    )
    

    The build then fails with the following error message:

    $ cmake -H/path/to/source-dir -B/path/to/build-dir
    ...
    $ cmake --build /path/to/build-dir/ --target all
    ...
    CMake Error: The source directory "/path/to/build-dir/EP_libfoo/src/libfoo" does not appear to contain CMakeLists.txt.
    ...
    $
    

    So, as pointed out in the above directory layout, CMake thinks that the root of
    the external project is

    /path/to/build-dir/EP_libfoo/src/libfoo
    

    when, in fact, it is

    /path/to/build-dir/EP_libfoo/src/libfoo/libfoo
    

    My attempts to solve this problem:

    1. Unfortunately, changing the argument SOURCE_DIR of ExternalProject did
      not work, because the value of this variable is used as the location to
      which the git repository of libfoo is cloned into. This results in a recursive dependency hell which cannot be broken.

    2. Changing the directory layout of libfoo to comply with ExternalProject.
      Obviously, this would work but it might not work for other (read-only)
      third party libraries.

    3. Abusing the update/patch step of ExternalProject, e.g. by specifying

      set( EP_LIBFOO_DIR "${CMAKE_CURRENT_BINARY_DIR}/EP_libfoo" )
      
      ExternalProject_Add( libfoo 
          PREFIX            "${EP_LIBFOO_DIR}"
          GIT_REPOSITORY    "<link to remote which hosts libfoo.git>" 
          GIT_TAG           "<some hash>"
      
          # Copy the content of `<...>/libfoo/libfoo` into `<...>/libfoo`.
          # Note to self: using symlinks instead copying is too platform-specific.
          PATCH_COMMAND     ${CMAKE_COMMAND} -E copy_directory "${EP_LIBFOO_DIR}/src/libfoo/libfoo" "${EP_LIBFOO_DIR}/src/libfoo"
      )
      

      This works but it’s hackish and very prone to fail with other external projects.

    4. Building on the solution to another problem: add a temporary
      CMakeLists.txt in the location where CMake assumes it. This temporary file
      then includes the actual CMakeLists.txt:

      set( EP_LIBFOO_DIR "${CMAKE_CURRENT_BINARY_DIR}/EP_libfoo" )
      set( GENERATED_DIR "${CMAKE_BINARY_DIR}/generated" )
      
      file( MAKE_DIRECTORY ${GENERATED_DIR} )
      file( WRITE ${GENERATED_DIR}/CMakeLists.txt
          "cmake_minimum_required( VERSION 3.0 )\n"
          "add_subdirectory( libfoo )\n" 
      )
      
      ExternalProject_Add( libfoo 
          PREFIX            "${EP_LIBFOO_DIR}"
          GIT_REPOSITORY    "<link to remote which hosts libfoo.git>" 
          GIT_TAG           "<some hash>"
      
          # Copy the 
          UPDATE_COMMAND    ${CMAKE_COMMAND} -E copy ${CMAKE_CURRENT_BINARY_DIR}/generated/CMakeLists.txt ${EP_LIBFOO_DIR}/src/libfoo
      )
      

      This works as well and feels better than the previous solution.

    However, does a more elegant exist to do the same?

  • Is it possible to clone a git repository, and split the clone repo, keeping track of the changes from original repo?
  • Couldn't find any revision to build. Verify the repository and branch configuration for this job. Finished: FAILURE
  • maven Error resolving version for plugin maven-eclipse-plugin in Jenkins
  • Solution/Project name variable in TFS Build 2010
  • Create GIT tags inside Bamboo
  • How schedule build in Jenkins?
  • 6 Solutions collect form web for “CMake ExternalProject: how to specify relative path to the root CMakeLists.txt?”

    I’ve submitted a merge request to add a SOURCE_SUBDIR option to ExternalProject_Add that will solve this use case. Hopefully it will be available in CMake 3.7. (You can also copy ExternalProject*.cmake locally into your own project to take advantage of the feature immediately.)

    You can use

    SOURCE_DIR /path/to/build-dir/EP_libfoo/src/libfoo/libfoo  
    

    ExternalProject_Add call. That designates the actual source directory.

    You can simply overwrite cmake command ant set path to the CMakeLists.txt manually. For example

    ExternalProject_Add(libfoo
       PREFIX            "${EP_LIBFOO_DIR}"
       GIT_REPOSITORY    "<link to remote which hosts libfoo.git>" 
       GIT_TAG           "<some hash>"
       CONFIGURE_COMMAND ${CMAKE_COMMAND} -G${CMAKE_GENERATOR} "${EP_LIBFOO_DIR}/src/libfoo/libfoo")
    

    I’ve been struggling with the same issue in a project that I am working on and this is the solution I have been able to come up with. It results in not using ExternalProject for the git handling but results in the same behavior as far as I have been able to tell.

    CMakeLists.txt

    include(ExternalProject)
    set(libfoo_prefix ${CMAKE_HOME_DIRECTORY}/libfoo)
    
    # during generation remove any previous repo and clone.
    file(REMOVE_RECURSE ${libfoo_prefix})
    execute_process(
        COMMAND git clone <link to remote which hosts libfoo.git>
        WORKING_DIRECTORY ${CMAKE_HOME_DIRECTORY})
    
    # Add the external project.
    ExternalProject_Add(libfoo
        PREFIX ${libfoo_prefix}
        SOURCE_DIR ${libfoo_prefix}/libfoo)
    # As part of the pre-build step update the git repo.
    add_custom_command(
        TARGET libfoo
        PRE_BUILD
        COMMAND ${CMAKE_COMMAND} -P GitPull.cmake)
    

    GitPull.cmake

    execute_process(
        COMMAND git pull origin master
        WORKING_DIRECTORY ${CMAKE_SOURCE_DIRECTORY}/libfoo)
    

    Here is a simple solution

    set(EXTERNAL_PROJECTS "")
    set(EXTERNAL_LIBS "")
    include(ExternalProject)
    
    # Set compiler(s) per project as required to CMAKE_ARGS in ExternalProject_Add(..).
    #       -DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}
    #       -DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}
    set(EXTERNAL_CMAKE_ARGS -D CMAKE_SYSROOT=${CMAKE_SYSROOT}
        -DCMAKE_FIND_ROOT_PATH_MODE_PROGRAM=${CMAKE_FIND_ROOT_PATH_MODE_PROGRAM}
        -DCMAKE_FIND_ROOT_PATH_MODE_LIBRARY=${CMAKE_FIND_ROOT_PATH_MODE_LIBRARY}
        -DCMAKE_FIND_ROOT_PATH_MODE_INCLUDE=${CMAKE_FIND_ROOT_PATH_MODE_INCLUDE}
        -DCMAKE_FIND_ROOT_PATH_MODE_PACKAGE=${CMAKE_FIND_ROOT_PATH_MODE_PACKAGE}
    )
    
    set(AIOUSB aiousb)
    set(AIOUSB_SRC aiousb_src)
    set(EXTERNAL_PROJECTS ${EXTERNAL_PROJECTS} ${AIOUSB_SRC} ${AIOUSB})
    set(AIOUSB_SRC_GIT_BRANCH "master")
    
    ExternalProject_Add(${AIOUSB_SRC}
        PREFIX ${AIOUSB_SRC}
        GIT_REPOSITORY "https://github.com/accesio/AIOUSB.git"
        GIT_TAG ${AIOUSB_SRC_GIT_BRANCH}
        UPDATE_COMMAND ""
        CONFIGURE_COMMAND ""
        BUILD_COMMAND ""
        BUILD_BYPRODUCTS ${CMAKE_BINARY_DIR}/${AIOUSB_SRC}/src/${AIOUSB_SRC}/AIOUSB/CMakeLists.txt
        INSTALL_COMMAND ""
    )
    
    set(AIOUSB_LIBRARY ${CMAKE_BINARY_DIR}/${AIOUSB}/src/${AIOUSB}-build/lib/${CMAKE_STATIC_LIBRARY_PREFIX}${AIOUSB}${CMAKE_STATIC_LIBRARY_SUFFIX})
    set(AIOUSBCPP_LIBRARY ${CMAKE_BINARY_DIR}/${AIOUSB}/src/${AIOUSB}-build/lib/${CMAKE_STATIC_LIBRARY_PREFIX}${AIOUSB}cpp${CMAKE_STATIC_LIBRARY_SUFFIX})
    
    ExternalProject_Add(${AIOUSB}
        DEPENDS ${AIOUSB_SRC}
        PREFIX ${AIOUSB}
        DOWNLOAD_COMMAND ""
        SOURCE_DIR ${CMAKE_BINARY_DIR}/${AIOUSB_SRC}/src/${AIOUSB_SRC}/AIOUSB
        CMAKE_ARGS ${EXTERNAL_CMAKE_ARGS} -DCMAKE_C_COMPILER=${CMAKE_C_COMPILER} -DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}
            -DBUILD_SAMPLES:BOOL=OFF -DBUILD_AIOUSB_SHARED:BOOL=OFF -DBUILD_AIOUSBDBG_SHARED:BOOL=OFF -DBUILD_AIOUSBCPP_SHARED:BOOL=OFF
            -DBUILD_AIOUSBCPPDBG_SHARED:BOOL=OFF
        BUILD_BYPRODUCTS ${CMAKE_SOURCE_DIR}/src/lib/include/${AIOUSB} ${CMAKE_BINARY_DIR}/${AIOUSB}/src/${AIOUSB}-build/lib/libaiousb.a
            ${CMAKE_BINARY_DIR}/${AIOUSB}/src/${AIOUSB}-build/lib/libaiousbcpp.a
        INSTALL_COMMAND rm -rf ${CMAKE_SOURCE_DIR}/src/lib/include/${AIOUSB} && mkdir -p ${CMAKE_SOURCE_DIR}/src/lib/include/${AIOUSB} &&
            echo "ln -sr ${CMAKE_BINARY_DIR}/${AIOUSB_SRC}/src/${AIOUSB_SRC}/AIOUSB/lib/*.h ${CMAKE_SOURCE_DIR}/src/lib/include/${AIOUSB}" | bash
    )
    
    set(LIBAIOUSB libaiousb)
    add_library(${LIBAIOUSB} STATIC IMPORTED)
    set_property(TARGET ${LIBAIOUSB} PROPERTY IMPORTED_LOCATION ${AIOUSB_LIBRARY})
    add_dependencies(${LIBAIOUSB} ${AIOUSB})
    set(EXTERNAL_LIBS ${EXTERNAL_LIBS} ${LIBAIOUSB})
    
    set(LIBAIOUSBCPP libaiousbcpp)
    add_library(${LIBAIOUSBCPP} STATIC IMPORTED)
    set_property(TARGET ${LIBAIOUSBCPP} PROPERTY IMPORTED_LOCATION ${AIOUSBCPP_LIBRARY})
    add_dependencies(${LIBAIOUSBCPP} ${AIOUSB})
    set(EXTERNAL_LIBS ${EXTERNAL_LIBS} ${LIBAIOUSBCPP})
    
    ...
    add_dependencies(${PROJECT_NAME} ${EXTERNAL_PROJECTS})
    ...
    also add 
    target_link_libraries(${PROJECT_NAME} ${EXTERNAL_LIBS} ...)
    

    Basically you break it into two parts. The first just to get the source and the second to buld the software. I manually create links to the headers so kdevelop’s parser does not have to parse the entire project.

    For those, who still looking for an answer: try to specify CONFIGURE_COMMAND:

    ExternalProject_Add(libfoo 
        GIT_REPOSITORY "<link to remote which hosts libfoo.git>" 
        GIT_TAG "<some hash>"
        SOURCE_DIR "where to put the source"
        CONFIGURE_COMMAND
            "${CMAKE_COMMAND}"
            "-HPathToDirectoryWhereTheCMakeListsAre"
            "-BWhereToBuild"
        BUILD_COMMAND
            "${CMAKE_COMMAND}" --build "Path to the directory where you are building (specified with -B flag in CONFIGURE_COMMAND)"
    )
    
    Git Baby is a git and github fan, let's start git clone.