bazel GPU build error with fatal error: external/nccl_archive/src/nccl.h: No such file or directory

We are trying to build Tensorflow Serving 0.5.1 with TensorFlow 1.0.0@07bb8ea

Basing on CUDA 7.5, cuDNN 5.
Bazel 0.4.4

cd serving && bazel build -c opt --config=cuda tensorflow_serving/...

ERROR: /root/.cache/bazel/_bazel_root/f8d1071c69ea316497c31e40fe0160
8c/external/org_tensorflow/tensorflow/contrib/nccl/BUILD:23:1: C++ c
ompilation of rule '@org_tensorflow//tensorflow/contrib/nccl:python/
ops/_nccl_ops.so' failed: crosstool_wrapper_driver_is_not_gcc failed
: error executing command external/local_config_cuda/crosstool/clang
/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE '-D_FORTI
FY_SOURCE=1' -fstack-protector -fPIE -Wall -Wunused-but-set-paramete
r ... (remaining 76 argument(s) skipped): com.google.devtools.build.
lib.shell.BadExitStatusException: Process exited with status 1.
In file included from external/org_tensorflow/tensorflow/contrib/ncc
l/kernels/nccl_manager.cc:15:0:
external/org_tensorflow/tensorflow/contrib/nccl/kernels/nccl_manager
.h:23:44: fatal error: external/nccl_archive/src/nccl.h: No such fil
e or directory
 #include "external/nccl_archive/src/nccl.h"
                                            ^
compilation terminated.
INFO: Elapsed time: 147.378s, Critical Path: 107.11s

I'm able to find nccl.h, but it can't be found during bazel build. Any suggestions? Thanks in advanced.

find / -name nccl.h
/root/.cache/bazel/_bazel_root/5071e8dca1385fb776f72b33971bf157/exte
rnal/nccl_archive/src/nccl.h
/root/.cache/bazel/_bazel_root/f8d1071c69ea316497c31e40fe01608c/exte
rnal/nccl_archive/src/nccl.h

Same error here.

cuda 8.0
cudnn 5.1
bazel 4.2

ERROR: /root/.cache/bazel/_bazel_root/f8d1071c69ea316497c31e40fe01608c/external/org_tensorflow/tensorflow/contrib/nccl/BUILD:23:1: C++ compilation of rule '@org_tensorflow//tensorflow/contrib/nccl:python/ops/_nccl_ops.so' failed: crosstool_wrapper_driver_is_not_gcc failed: error executing command external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -fPIE -Wall -Wunused-but-set-parameter ... (remaining 77 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
In file included from external/org_tensorflow/tensorflow/contrib/nccl/kernels/nccl_manager.cc:15:0:
external/org_tensorflow/tensorflow/contrib/nccl/kernels/nccl_manager.h:23:44: fatal error: external/nccl_archive/src/nccl.h: No such file or directory
compilation terminated.

Any solutions?

@ kirilg，can you help take a quick look at this issue? Thank you.

same here

To get around it you can comment out the DEP for nccl in: tensorflow/tensorflow/contrib/BUILD

Line 42 iirc

@jlertle

Thanks, @jlertle

@jlertle

Thanks @jlertle.

which line in: tensorflow/tensorflow/contrib/BUILD is the DEP for nccl? i can't find it, thanks.

65: "//tensorflow/contrib/nccl:nccl_py",

I believe...

It was moved into a Windows check but the referenced path is still having issues resolving during Serving build process on Ubuntu. Bazel stuff.

I tried a script provided by #318, it works fine

If you comment it out examples fail, I managed to built it as well but... I get
ImportError: cannot import name nccl with a minst example.

Here is the task that fails:

>>>>> # @org_tensorflow//tensorflow/contrib/nccl:python/ops/_nccl_ops.so [action 'Compiling external/org_tensorflow/tensorflow/contrib/nccl/kernels/nccl_manager.cc']
(cd /root/.cache/bazel/_bazel_root/f8d1071c69ea316497c31e40fe01608c/execroot/serving && \
  exec env - \
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE 
  '-D_FORTIFY_SOURCE=1' -fstack-protector -fPIE -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object
   -fno-omit-frame-pointer -g0 -O2 -DNDEBUG -ffunction-sections -fdata-sections '-std=c++11' -MD 
   -MF bazel-out/local_linux-opt/bin/external/org_tensorflow/tensorflow/contrib/nccl/_objs/python/ops/_nccl_ops.so/external/org_tensorflow/tensorflow/contrib/nccl/kernels/nccl_manager.pic.d
    '-frandom-seed=bazel-out/local_linux-opt/bin/external/org_tensorflow/tensorflow/contrib/nccl/_objs/python/ops/_nccl_ops.so/external/org_tensorflow/tensorflow/contrib/nccl/kernels/nccl_manager.pic.o' -fPIC -DEIGEN_MPL2_ONLY 
  -iquote external/org_tensorflow -iquote bazel-out/local_linux-opt/genfiles/external/org_tensorflow -iquote external/bazel_tools 
  -iquote bazel-out/local_linux-opt/genfiles/external/bazel_tools -iquote external/nccl_archive 
  -iquote bazel-out/local_linux-opt/genfiles/external/nccl_archive -iquote external/local_config_cuda 
  -iquote bazel-out/local_linux-opt/genfiles/external/local_config_cuda -iquote external/protobuf 
  -iquote bazel-out/local_linux-opt/genfiles/external/protobuf -iquote external/eigen_archive 
  -iquote bazel-out/local_linux-opt/genfiles/external/eigen_archive -iquote external/local_config_sycl
   -iquote bazel-out/local_linux-opt/genfiles/external/local_config_sycl -isystem external/bazel_tools/tools/cpp/gcc3 
   -isystem external/local_config_cuda/cuda -isystem bazel-out/local_linux-opt/genfiles/external/local_config_cuda/cuda
    -isystem external/local_config_cuda/cuda/include -isystem bazel-out/local_linux-opt/genfiles/external/local_config_cuda/cuda/include 
    -isystem external/protobuf/src -isystem bazel-out/local_linux-opt/genfiles/external/protobuf/src -isystem external/eigen_archive 
    -isystem bazel-out/local_linux-opt/genfiles/external/eigen_archive -DEIGEN_AVOID_STL_ARRAY -Iexternal/gemmlowp -Wno-sign-compare -fno-exceptions '-DGOOGLE_CUDA=1' -msse3 -pthread -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -fno-canonical-system-headers -c external/org_tensorflow/tensorflow/contrib/nccl/kernels/nccl_manager.cc -o bazel-out/local_linux-opt/bin/external/org_tensorflow/tensorflow/contrib/nccl/_objs/python/ops/_nccl_ops.so/external/org_tensorflow/tensorflow/contrib/nccl/kernels/nccl_manager.pic.o)
ERROR: /root/.cache/bazel/_bazel_root/f8d1071c69ea316497c31e40fe01608c/external/org_tensorflow/tensorflow/contrib/nccl/BUILD:23:1: C++ compilation of rule '@org_tensorflow//tensorflow/contrib/nccl:python/ops/_nccl_ops.so' failed: crosstool_wrapper_driver_is_not_gcc failed: error executing command external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -fPIE -Wall -Wunused-but-set-parameter ... (remaining 77 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
In file included from external/org_tensorflow/tensorflow/contrib/nccl/kernels/nccl_manager.cc:15:0:
external/org_tensorflow/tensorflow/contrib/nccl/kernels/nccl_manager.h:23:44: fatal error: external/nccl_archive/src/nccl.h:
 No such file or directory
 #include "external/nccl_archive/src/nccl.h"
                                            ^

I verified that nccl_Archive is fetched and unzipped correctly under .cache dir and from what I see
-iquote external/nccl_archive should do the work to include all stuff needed.

I solved it by removing the prefix /external/nccl_archive.

@skonto

@skonto removing prefix /external/nccl_archive in files nccl_ops.cc and
nccl_manager.h which in folder tensorflow/tensorflow/contrib/nccl/kernels, fix the issue

git clone https://github.com/NVIDIA/nccl.git
cd nccl/
make CUDA_HOME=/usr/local/cuda

sudo make install
sudo mkdir -p /usr/local/include/external/nccl_archive/src
sudo ln -s /usr/local/include/nccl.h /usr/local/include/external/nccl_archive/src/nccl.h

Tell me what version of Cuda do you have and what version of Nccl do you have and also any thing else I can replicate your environtment.

cudda 9, and latest nccl

How did you set up your nccl2 ?

@cyberwillis

@cyberwillis
nccl-install-guide
which is the install link,

https://blog.csdn.net/hanchan94/article/details/80265922

@cjhkeep

@cjhkeep
I dont have any problem with nccl.h 1.3 or 2.xx at all. I don't need a guide

However if you install "NCCL2" using the ".deb" version your nccl.h will be far away from where tensorflow expect to find it! That's why a suggested installing from the tar file here .

I was trying to diagnose the problem of our colleague above, asking him what the process he had installed nccl on his machine.

[UPDATED]

Wouldn't it be the right way to tensorflow to just look at the right directories? /usr/include/ is the place for header files in linux, I don't get why it looks somewhere else..?

NVIDIA in times to times change the locations of its packages (because they think its funny) :)
If you investigate a little, depending on your cuda version some files go to some places others go to another places... I believe NVidia doesn't have a stable ideia where to put this things exacly and tensorflow cannot enter on their hell.

Closing - please see the latest Docker examples for bringing up a build environment. The GPU build addresses the NCCL dependency.

@gautamvasudevan

@gautamvasudevan This is still a bug when trying to do macOS GPU builds. Since your Docker example doesn't work on Mac, I think this issue should still be open.

@praeclarum

Hi @praeclarum I am sorry to see your question only now, sadlly I am not using Tensorflow anymore, but I believe that since the Docker Tensorflow rely on the abstract install from Ubuntu you can change it for the exact problem you having. Can you post exact what problem are you getting on your MAC ?

Another question... If you are using Docker on your Mac to build TFServing... how do you make the GPU Passthrow to Docker Engine ? Since the X11 forwarding does not exist on MacOS (instead apple uses Quartz). So your GPU will never be recognized inside docker because Apple does not allow it formally.

I believe your only strategy is to translate the commands from Docker file into commands in your Homebrew.

In case you really want to try make the GPU available inside docker (macOS only) you can try use XQuartz take a look on this Gist https://gist.github.com/cschiewek/246a244ba23da8b9f0e7b11a68bf3285

We don't have any official support for macOS and nccl builds currently, though feel free to file a new issue specifically for macOS, we welcome any community support here!

seems that now there's a --config=nonccl option you can add to a bazel command, e.g. bazel build --config=opt --config=cuda --config=nonccl //tensorflow/tools/pip_package:build_pip_package (dunno if this will work entirely, but it seems to get me past this error ...)

@gatoatigrado

@gatoatigrado thanks for the tip, that's exactly what I was looking for

Navigation Menu

Repository navigation

bazel GPU build error with fatal error: external/nccl_archive/src/nccl.h: No such file or directory #327

16 remaining items

@cyberwillis
nccl-install-guide
which is the install link,

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

bazel GPU build error with fatal error: external/nccl_archive/src/nccl.h: No such file or directory #327

Description

Activity

tvkpz commented on Feb 19, 2017

cheyang commented on Feb 23, 2017

kinhunt commented on Feb 24, 2017

jlertle commented on Feb 24, 2017

cheyang commented on Feb 25, 2017

sskgit commented on Feb 25, 2017

cosastro commented on Mar 24, 2017

perdasilva commented on Mar 24, 2017

jlertle commented on Mar 24, 2017

cosastro commented on Mar 27, 2017

skonto commented on Apr 10, 2017

skonto commented on Apr 11, 2017

Lukeisme commented on Apr 14, 2017

perdasilva commented on Jun 2, 2017

16 remaining items

cyberwillis commented on May 12, 2018

rickragv commented on May 12, 2018

cyberwillis commented on May 12, 2018

cjhkeep commented on May 14, 2018

@cyberwillis nccl-install-guide which is the install link,

cyberwillis commented on May 14, 2018

discordianfish commented on May 31, 2018

cyberwillis commented on May 31, 2018

gautamvasudevan commented on Jun 29, 2018

praeclarum commented on Sep 10, 2018

cyberwillis commented on Sep 10, 2018

cyberwillis commented on Sep 10, 2018

gautamvasudevan commented on Sep 10, 2018

gatoatigrado commented on Dec 12, 2018

praeclarum commented on Dec 13, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

@cyberwillis
nccl-install-guide
which is the install link,