Skip to content

Compile bug: Vulkan shaders build fails due to missing vulkan-shaders directory during ExternalProject\_Add configure step #13753

Closed
Closed
Compile bug: Vulkan shaders build fails due to missing vulkan-shaders directory during ExternalProject\_Add configure step#13753
Assignees
mtmcp
Labels
bug-unconfirmedstale

Description

@Manamama
Manamama
opened on May 24, 2025

Git commit

Termux again. ChatGPT wrote most of below:

Title: Vulkan shaders build fails due to missing vulkan-shaders directory during ExternalProject_Add configure step

Description:

When building with -DGGML_VULKAN=ON, the build fails during the vulkan-shaders-gen ExternalProject configuration step with this error:

[  3%] Built target ggml-base
[  4%] Creating directories for 'vulkan-shaders-gen'
[  4%] No download step for 'vulkan-shaders-gen'
[  5%] No update step for 'vulkan-shaders-gen'
[  5%] No patch step for 'vulkan-shaders-gen'
[  5%] Performing configure step for 'vulkan-shaders-gen'
-- The C compiler identification is Clang 20.1.5
-- The CXX compiler identification is Clang 20.1.5
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /data/data/com.termux/files/usr/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /data/data/com.termux/files/usr/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at CMakeLists.txt:3 (add_subdirectory):
  add_subdirectory given source "vulkan-shaders" which is not an existing
  directory.

Workaround:

Manual compilation of vulkan-shaders-gen is required:

  1. Manually build vulkan-shaders-gen inside
    /data/data/com.termux/files/home/downloads/llama.cpp/ggml/src/ggml-vulkan/vulkan-shaders/

  2. Copy the executable to the expected binary directory before resuming the main build:

cp /data/data/com.termux/files/home/downloads/llama.cpp/ggml/src/ggml-vulkan/vulkan-shaders/bin/vulkan-shaders-gen bin/

Likely culprit:

The problem appears to originate from the CMake setup in the Vulkan backend, specifically either:

  • ggml/src/CMakeLists.txt around the ExternalProject_Add(vulkan-shaders-gen ...)
  • or ggml/src/ggml-vulkan/CMakeLists.txt which handles shader building and the vulkan-shaders subdirectory references.

The build system tries to add_subdirectory("vulkan-shaders") in a path where that folder is missing, causing the configure step failure.

Request:

Please review the Vulkan shader CMake setup to ensure that:

  • The vulkan-shaders source directory exists or is properly downloaded/copied before the configure step.

  • The ExternalProject_Add commands correctly reference existing paths, avoiding the manual compile-and-copy workaround.

My take: it compiles with the bells and whistles on Termux: cmake -S . -B build -DLLAMA_CURL=ON -DGGML_VULKAN=ON -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS -DGGML_CUDA=OFF -DGGML_ARCH_FLAGS="-march=armv8.2-a+dotprod+fp16" , but these resulting executables sometimes need hunting for by hand, due to some weird:

~/.../llama.cpp/build $ ls /data/data/com.termux/files/home/downloads/llama.cpp/build/bin/vulkan-shaders-gen
lsd: /data/data/com.termux/files/home/downloads/llama.cpp/build/bin/vulkan-shaders-gen: No such file or directory (os error 2).

~/.../llama.cpp/build $ cp /data/data/com.termux/files/home/downloads/llama.cpp/ggml/src/ggml-vulkan/vulkan-shaders/bin/vulkan-shaders-gen /data/data/com.termux/files/home/downloads/llama.cpp/build/bin/vulkan-shaders-gen
~/.../llama.cpp/build $ ls /data/data/com.termux/files/home/downloads/llama.cpp/build/bin/vulkan-shaders-gen
 /data/data/com.termux/files/home/downloads/llama.cpp/build/bin/vulkan-shaders-gen
~/.../llama.cpp/build $ make install 
[  3%] Built target ggml-base
[  6%] Built target vulkan-shaders-gen-build
[  6%] Performing install step for 'vulkan-shaders-gen'
-- Install configuration: "RelWithDebInfo"
CMake Error at cmake_install.cmake:52 (file):
  file INSTALL cannot find
  "/data/data/com.termux/files/home/downloads/llama.cpp/build/bin/vulkan-shaders-gen":
  No such file or directory.


make[2]: *** [ggml/src/ggml-vulkan/CMakeFiles/vulkan-shaders-gen-install.dir/build.make:73: ggml/src/ggml-vulkan/vulkan-shaders-gen-prefix/src/vulkan-shaders-gen-stamp/vulkan-shaders-gen-install] Error 1
make[1]: *** [CMakeFiles/Makefile2:2390: ggml/src/ggml-vulkan/CMakeFiles/vulkan-shaders-gen-install.dir/all] Error 2
make: *** [Makefile:146: all] Error 2
~/.../llama.cpp/build $ make install 
[  3%] Built target ggml-base
[  6%] Built target vulkan-shaders-gen-build
[  6%] Performing install step for 'vulkan-shaders-gen'
-- Install configuration: "RelWithDebInfo"
CMake Error at cmake_install.cmake:52 (file):
  file INSTALL cannot find
  "/data/data/com.termux/files/home/downloads/llama.cpp/build/bin/vulkan-shaders-gen":
  No such file or directory.

Operating systems

Other? (Please let us know in description)

GGML backends

Vulkan

Problem description & steps to reproduce

See above

First Bad Commit

No response

Compile command

cmake -S . -B build -DLLAMA_CURL=ON -DGGML_VULKAN=ON

Relevant log output

See above

Activity

  1. Manamama
    added
    bug-unconfirmed
    on May 24, 2025

  2. Manamama commented on May 24, 2025

    @Manamama
    Manamama
    on May 24, 2025
    Author

    Update, even the sudo trick to make it stay did not work:

    ~/.../llama.cpp/build $ sudo chmod 0555 /data/data/com.termux/files/home/downloads/llama.cpp/build/bin/vulkan-shaders-gen
Using: /data/data/com.termux/files/usr/bin/su ...
chmod: changing permissions of '/data/data/com.termux/files/home/downloads/llama.cpp/build/bin/vulkan-shaders-gen': Operation not permitted
~/.../llama.cpp/build $ sudo ls -l  /data/data/com.termux/files/home/downloads/llama.cpp/build/bin/vulkan-shaders-gen
Using: /data/data/com.termux/files/usr/bin/su ...
-rwx------. 1 u0_a278 u0_a278 2294584 May 24 16:43 /data/data/com.termux/files/home/downloads/llama.cpp/build/bin/vulkan-shaders-gen
~/.../llama.cpp/build $ sudo file   /data/data/com.termux/files/home/downloads/llama.cpp/build/bin/vulkan-shaders-gen
Using: /data/data/com.termux/files/usr/bin/su ...
/data/data/com.termux/files/home/downloads/llama.cpp/build/bin/vulkan-shaders-gen: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /system/bin/linker64, for Android 24, built by NDK r27c (12479018), with debug_info, not stripped

    so this trick advised by ChatGPT seems to work:

    ~/.../llama.cpp/build $ mkdir -p ~/bin_override
cp /data/data/com.termux/files/home/downloads/llama.cpp/ggml/src/ggml-vulkan/vulkan-shaders/bin/vulkan-shaders-gen ~/bin_override/
~/.../llama.cpp/build $ export PATH=$HOME/bin_override:$PATH
~/.../llama.cpp/build $ make install
[  3%] Built target ggml-base
[  6%] Built target vulkan-shaders-gen-build
[  6%] Performing install step for 'vulkan-shaders-gen'
-- Install configuration: "RelWithDebInfo"
-- Installing: /data/data/com.termux/files/home/downloads/llama.cpp/build/ggml/src/ggml-vulkan/vulkan-shaders-gen-prefix/src/vulkan-shaders-gen-build/bin/vulkan-shaders-gen
[  6%] Built target vulkan-shaders-gen-install
[  6%] Completed 'vulkan-shaders-gen'
[  6%] Built target vulkan-shaders-gen
[  7%] Generate vulkan shaders
ggml_vulkan: Generating and compiling shaders to SPIR-V
[  7%] Building CXX object ggml/src/ggml-vulkan/CMakeFiles/ggml-vulkan.dir/ggml-vulkan.cpp.o
/data/data/com.termux/files/home/downloads/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:1656:2: warning: extra ';' outside of a function is incompatible with C++98 [-Wc++98-compat-extra-semi]
 1656 | };
      |  ^
...

    at least until the next quirky Vulkan error, in turn.

  3. 0cc4m commented on May 26, 2025

    @0cc4m
    0cc4m
    on May 26, 2025
    Contributor

    In general, using ChatGPT to explain problems to you is fine, but please don't clutter the issue with its outputs. It means people have to take your issue apart and figure out which parts are relevant and which are hallucinated or irrelevant.

  4. 0cc4m
    assigned
    mtmcp
    on May 26, 2025

  5. Manamama commented on May 26, 2025

    @Manamama
    Manamama
    on May 26, 2025
    Author

    Re:

    In general, using ChatGPT to explain problems to you is fine, but please don't clutter the issue with its outputs. It means people have to take your issue apart and figure out which parts are relevant and which are hallucinated or irrelevant.

    I will try. Still, LLMs write such tickets better than me as they are pithier in fact. See this self-quote: "My take: it compiles with the bells and whistles on Termux ... but these resulting executables sometimes need hunting for by hand", etc.

    Ad rem: when trying to overcome the missing -sys switch in the experimental (see for now here: https://huggingface.co/ggml-org/ultravox-v0_5-llama-3_2-1b-GGUF/discussions/3#68343c596319d63c09592e26) , I have retried compilation with VULKAN on my Droid, and this bug persists:

    git pull
remote: Enumerating objects: 97, done.
remote: Counting objects: 100% (78/78), done.
remote: Compressing objects: 100% (50/50), done.
remote: Total 97 (delta 37), reused 41 (delta 28), pack-reused 19 (from 3)
Unpacking objects: 100% (97/97), 459.55 KiB | 66.00 KiB/s, done.
From https://github.com/ggml-org/llama.cpp
   2f099b51..22229314  master                        -> origin/master
   f2d876ad..b17e9811  cisc/jina-embeddings-v3       -> origin/cisc/jina-embeddings-v3
 * [new branch]        gg/embedding-fix-non-embd-use -> origin/gg/embedding-fix-non-embd-use
   118f9744..3ef770f9  gg/kv-cache-simplify-part3    -> origin/gg/kv-cache-simplify-part3
 * [new branch]        gg/vmm-fix-link               -> origin/gg/vmm-fix-link
 * [new tag]           b5488                         -> b5488
 * [new tag]           b5489                         -> b5489
 * [new tag]           b5490                         -> b5490
Updating 2f099b51..22229314

    ->

    ...
-- Adding CPU backend variant ggml-cpu: -mcpu=native+dotprod+noi8mm+nosve+nosme 
-- Found Vulkan: /data/data/com.termux/files/usr/lib/libvulkan.so (found version "1.4.315") found components: glslc glslangValidator
-- Vulkan found
-- GL_KHR_cooperative_matrix supported by glslc
-- GL_NV_cooperative_matrix2 not supported by glslc
-- GL_EXT_integer_dot_product not supported by glslc
-- GL_EXT_bfloat16 not supported by glslc
-- Including Vulkan backend
...
- Generating done (0.1s)
-- Build files have been written to: /data/data/com.termux/files/home/downloads/llama.cpp/build/ggml/src/ggml-vulkan/vulkan-shaders-gen-prefix/src/vulkan-shaders-gen-build
[41/248] Performing build step for 'vulkan-shaders-gen'
[1/2] Building CXX object CMakeFiles/vulkan-shaders-gen.dir/vulkan-shaders-gen.cpp.o
[2/2] Linking CXX executable /data/data/com.termux/files/home/downloads/llama.cpp/build/bin/vulkan-shaders-gen
[42/248] Performing install step for 'vulkan-shaders-gen'
FAILED: ggml/src/ggml-vulkan/vulkan-shaders-gen-prefix/src/vulkan-shaders-gen-stamp/vulkan-shaders-gen-install /data/data/com.termux/files/home/downloads/llama.cpp/build/ggml/src/ggml-vulkan/vulkan-shaders-gen-prefix/src/vulkan-shaders-gen-stamp/vulkan-shaders-gen-install 
cd /data/data/com.termux/files/home/downloads/llama.cpp/build/ggml/src/ggml-vulkan/vulkan-shaders-gen-prefix/src/vulkan-shaders-gen-build && /data/data/com.termux/files/usr/bin/cmake --install . && /data/data/com.termux/files/usr/bin/cmake -E touch /data/data/com.termux/files/home/downloads/llama.cpp/build/ggml/src/ggml-vulkan/vulkan-shaders-gen-prefix/src/vulkan-shaders-gen-stamp/vulkan-shaders-gen-install
-- Install configuration: "RelWithDebInfo"
CMake Error at cmake_install.cmake:52 (file):
  file INSTALL cannot find
  "/data/data/com.termux/files/home/downloads/llama.cpp/build/bin/vulkan-shaders-gen":

    Works in non-Droid Ubuntu, etc.

    Let me know if you need the output of e.g. vulkaninfo or termux-info.

  6. Manamama commented on May 26, 2025

    @Manamama
    Manamama
    on May 26, 2025
    Author

    Found a fix, replace ggml/src/ggml-vulkan/CMakeLists.txt with a version of below:

    cmake_minimum_required(VERSION 3.19)
cmake_policy(SET CMP0114 NEW)

find_package(Vulkan COMPONENTS glslc REQUIRED)
add_subdirectory(vulkan-shaders)

# Function to test shader extension support
function(test_shader_extension_support EXTENSION_NAME TEST_SHADER_FILE RESULT_VARIABLE)
    execute_process(
        COMMAND ${Vulkan_GLSLC_EXECUTABLE} -o - -fshader-stage=compute --target-env=vulkan1.3 "${TEST_SHADER_FILE}"
        OUTPUT_VARIABLE glslc_output
        ERROR_VARIABLE glslc_error
    )
    if (${glslc_error} MATCHES ".*extension not supported: ${EXTENSION_NAME}.*")
        message(STATUS "${EXTENSION_NAME} not supported by glslc")
        set(${RESULT_VARIABLE} OFF PARENT_SCOPE)
    else()
        message(STATUS "${EXTENSION_NAME} supported by glslc")
        set(${RESULT_VARIABLE} ON PARENT_SCOPE)
        add_compile_definitions(${RESULT_VARIABLE})
    endif()
endfunction()

if (Vulkan_FOUND)
    message(STATUS "Vulkan found")

    ggml_add_backend_library(ggml-vulkan
                             ggml-vulkan.cpp
                             ../../include/ggml-vulkan.h
                            )

    # Test all shader extensions
    test_shader_extension_support(
        "GL_KHR_cooperative_matrix"
        "${CMAKE_CURRENT_SOURCE_DIR}/vulkan-shaders/test_coopmat_support.comp"
        "GGML_VULKAN_COOPMAT_GLSLC_SUPPORT"
    )
    test_shader_extension_support(
        "GL_NV_cooperative_matrix2"
        "${CMAKE_CURRENT_SOURCE_DIR}/vulkan-shaders/test_coopmat2_support.comp"
        "GGML_VULKAN_COOPMAT2_GLSLC_SUPPORT"
    )
    test_shader_extension_support(
        "GL_EXT_integer_dot_product"
        "${CMAKE_CURRENT_SOURCE_DIR}/vulkan-shaders/test_integer_dot_support.comp"
        "GGML_VULKAN_INTEGER_DOT_GLSLC_SUPPORT"
    )
    test_shader_extension_support(
        "GL_EXT_bfloat16"
        "${CMAKE_CURRENT_SOURCE_DIR}/vulkan-shaders/test_bfloat16_support.comp"
        "GGML_VULKAN_BFLOAT16_GLSLC_SUPPORT"
    )

    target_link_libraries(ggml-vulkan PRIVATE Vulkan::Vulkan)
    target_include_directories(ggml-vulkan PRIVATE ${CMAKE_CURRENT_BINARY_DIR})

    # Workaround for clang-cl debug build
    if (MSVC AND CMAKE_CXX_COMPILER_ID STREQUAL "Clang")
        add_compile_definitions(_ITERATOR_DEBUG_LEVEL=0)
    endif()

    if (GGML_VULKAN_CHECK_RESULTS)
        add_compile_definitions(GGML_VULKAN_CHECK_RESULTS)
    endif()
    if (GGML_VULKAN_DEBUG)
        add_compile_definitions(GGML_VULKAN_DEBUG)
    endif()
    if (GGML_VULKAN_MEMORY_DEBUG)
        add_compile_definitions(GGML_VULKAN_MEMORY_DEBUG)
    endif()
    if (GGML_VULKAN_SHADER_DEBUG_INFO)
        add_compile_definitions(GGML_VULKAN_SHADER_DEBUG_INFO)
    endif()
    if (GGML_VULKAN_PERF)
        add_compile_definitions(GGML_VULKAN_PERF)
    endif()
    if (GGML_VULKAN_VALIDATE)
        add_compile_definitions(GGML_VULKAN_VALIDATE)
    endif()
    if (GGML_VULKAN_RUN_TESTS)
        add_compile_definitions(GGML_VULKAN_RUN_TESTS)
    endif()

    # Set up shader generation
    set(_ggml_vk_header ${CMAKE_CURRENT_BINARY_DIR}/ggml-vulkan-shaders.hpp)
    set(_ggml_vk_source ${CMAKE_CURRENT_BINARY_DIR}/ggml-vulkan-shaders.cpp)
    set(_ggml_vk_input_dir ${CMAKE_CURRENT_SOURCE_DIR}/vulkan-shaders)
    set(_ggml_vk_output_dir ${CMAKE_CURRENT_BINARY_DIR}/vulkan-shaders.spv)
    set(_ggml_vk_genshaders_cmd ${CMAKE_BINARY_DIR}/bin/vulkan-shaders-gen)

    file(GLOB _ggml_vk_shader_deps "${_ggml_vk_input_dir}/*.comp")
    add_custom_command(
        OUTPUT ${_ggml_vk_header} ${_ggml_vk_source}
        COMMAND ${_ggml_vk_genshaders_cmd}
            --glslc ${Vulkan_GLSLC_EXECUTABLE}
            --input-dir ${_ggml_vk_input_dir}
            --output-dir ${_ggml_vk_output_dir}
            --target-hpp ${_ggml_vk_header}
            --target-cpp ${_ggml_vk_source}
            --no-clean
        DEPENDS vulkan-shaders-gen ${_ggml_vk_shader_deps}
        COMMENT "Generate vulkan shaders"
    )

    target_sources(ggml-vulkan PRIVATE ${_ggml_vk_source} ${_ggml_vk_header})

else()
    message(WARNING "Vulkan not found")
endif()

    Here is quick why so, Grok AI this time:

    To resolve the conflict, I modified ggml/src/ggml-vulkan/CMakeLists.txt to:
    Use add_subdirectory(vulkan-shaders) to build vulkan-shaders-gen directly, assuming vulkan-shaders/CMakeLists.txt defines it.

    Remove ExternalProject_Add(vulkan-shaders-gen) and related cross-compilation logic to avoid the duplicate target error.

    Retain the shader generation logic (add_custom_command) to generate ggml-vulkan-shaders.hpp and ggml-vulkan-shaders.cpp.

    This approach simplifies the build process in Termux/proot environments, avoids complex cross-compilation, and ensures vulkan-shaders-gen is built and installed correctly.

    Proof, after that fix:

    /downloads/llama.cpp $ mv ggml/src/ggml-vulkan/CMakeLists.txt  ggml/src/ggml-vulkan/CMakeLists.txt.bak
~/downloads/llama.cpp $ nano ggml/src/ggml-vulkan/CMakeLists.txt
 ~/downloads/llama.cpp $ bash  ../_install_llama1.sh 
ver. 2.4
-- ccache found, compilation results will be cached. Disable with GGML_CCACHE=OFF.
-- CMAKE_SYSTEM_PROCESSOR: aarch64
-- Including CPU backend
-- ARM detected
-- ARM -mcpu not found, -mcpu=native will be used
-- ARM feature DOTPROD enabled
-- ARM feature FMA enabled
-- ARM feature FP16_VECTOR_ARITHMETIC enabled
-- Adding CPU backend variant ggml-cpu: -mcpu=native+dotprod+noi8mm+nosve+nosme 
-- BLAS found, Libraries: /data/data/com.termux/files/usr/lib/libopenblas.so
-- Checking for module 'openblas64'
--   No package 'openblas64' found
-- Checking for module 'openblas'
--   Found openblas, version 0.3.29
-- BLAS found, Includes: /data/data/com.termux/files/usr/include/openblas
-- Including BLAS backend
-- Vulkan found
-- GL_KHR_cooperative_matrix supported by glslc
-- GL_NV_cooperative_matrix2 not supported by glslc
-- GL_EXT_integer_dot_product not supported by glslc
-- GL_EXT_bfloat16 not supported by glslc
-- Including Vulkan backend
-- Configuring done (7.4s)
-- Generating done (0.8s)
-- Build files have been written to: /data/data/com.termux/files/home/downloads/llama.cpp/build
[  3%] Built target ggml-base
[  3%] Linking CXX executable ../../../../bin/vulkan-shaders-gen
[  4%] Built target vulkan-shaders-gen
[  4%] Generate vulkan shaders
ggml_vulkan: Generating and compiling shaders to SPIR-V
[  5%] Building CXX object ggml/src/ggml-vulkan/CMakeFiles/ggml-vulkan.dir/ggml-vulkan.cpp.o
/data/data/com.termux/files/home/downloads/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:1656:2: warning: extra ';' outside of a function is incompatible with C++98 [-Wc++98-compat-extra-semi]
 1656 | };
      |  ^

...

  7. mtmcp commented on May 26, 2025

    @mtmcp
    mtmcp
    on May 26, 2025
    Collaborator

    Hopefully that will work. But, we'll need you to submit a pull request containing the proposed fix and reference this issue from there. Chances are it might break the build for many of the CI builds.

    Please be sure to include all relevant build command-lines in addition to info on how your toolchain is set up. Many of us probably do not use this platform so it's a bit of guesswork on my part.

  8. github-actions
    added
    stale
    on Jun 26, 2025

  9. github-actions commented on Jul 10, 2025

    @github-actions
    github-actionsBot
    on Jul 10, 2025 – with GitHub Actions
    Contributor

    This issue was closed because it has been inactive for 14 days since being marked as stale.

  10. github-actions
    closed this as completedon Jul 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bug-unconfirmedstale

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions