GH-43416: [CI] Upgrade our vcpkg version on .env by raulcd · Pull Request #43417 · apache/arrow

raulcd · 2024-07-25T09:48:26Z

Rationale for this change

We are using a pretty old version and we are seeing some CI jobs.

What changes are included in this PR?

Upgrade vcpkg version

Are these changes tested?

on CI

Are there any user-facing changes?

No

GitHub Issue: [Python][CI][C++] Vcpkg failures building some wheels #43416

github-actions · 2024-07-25T09:48:50Z

⚠️ GitHub issue #43416 has been automatically assigned in GitHub to PR creator.

raulcd · 2024-07-25T09:54:53Z

@github-actions crossbow submit -g vcpkg wheel java-jars

github-actions · 2024-07-25T09:56:01Z

Unable to match any tasks for `wheel`
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/10092011041

raulcd · 2024-07-25T09:57:09Z

@github-actions crossbow submit -g vcpkg -g wheel java-jars

raulcd · 2024-07-25T10:53:25Z

@github-actions crossbow submit python-wheel-manylinux-2-28

github-actions · 2024-07-25T10:54:32Z

Unable to match any tasks for `python-wheel-manylinux-2-28`
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/10092813787

raulcd · 2024-07-25T10:59:37Z

@github-actions crossbow submit wheel-manylinux-2-28-cp39-amd64

github-actions · 2024-07-25T11:01:50Z

Revision: a60bc06affe1172480d60d7790c4536441f6e218

Submitted crossbow builds: ursacomputing/crossbow @ actions-243699443d

Task	Status
wheel-manylinux-2-28-cp39-amd64

raulcd · 2024-07-25T14:59:23Z

I am unsure why ORC is failing to build when calling fetchcontent_makeavailable(orc) with:

CMake Error at /tmp/arrow-build/_deps/orc-src/cmake_modules/ThirdpartyToolchain.cmake:467 (elseif):
  given arguments:

    "ORC_PREFER_STATIC_PROTOBUF" "AND"

  Unknown arguments specified
Call Stack (most recent call first):
  /tmp/arrow-build/_deps/orc-src/CMakeLists.txt:166 (INCLUDE)

I can see we define (since #43011):

    set(ORC_PREFER_STATIC_PROTOBUF
        OFF
        CACHE BOOL "" FORCE)

debugging locally I can see we use ORC 2.0.1 and the failure is coming from:

if (ORC_PACKAGE_KIND STREQUAL "conan")
  target_link_libraries (orc_protobuf INTERFACE ${protobuf_LIBRARIES})
elseif (ORC_PREFER_STATIC_PROTOBUF AND ${PROTOBUF_STATIC_LIB})
  target_link_libraries (orc_protobuf INTERFACE ${PROTOBUF_STATIC_LIB})
else ()
  target_link_libraries (orc_protobuf INTERFACE ${PROTOBUF_LIBRARY})
endif()

@kou any idea why fetchcontent building ORC might not see the defined ORC_PREFER_STATIC_PROTOBUF?

raulcd · 2024-07-26T10:02:23Z

@github-actions crossbow submit wheel-manylinux-2-28-cp39-amd64

raulcd · 2024-07-26T10:04:05Z

I am not sure why but applying the patch here: ee89232 is able to build orc

edited commit

github-actions · 2024-07-26T10:04:35Z

Revision: 982a6ce86f275253ced74bfdb3e37f2e5884f6fd

Submitted crossbow builds: ursacomputing/crossbow @ actions-46a723f68f

Task	Status
wheel-manylinux-2-28-cp39-amd64

raulcd · 2024-07-26T15:24:25Z

I am currently trying to understand why protoc fails on flight with SIGSEGV:

[143/176] Generating Flight.pb.cc, Flight.pb.h, Flight.grpc.pb.cc, Flight.grpc.pb.h
FAILED: src/arrow/flight/Flight.pb.cc src/arrow/flight/Flight.pb.h src/arrow/flight/Flight.grpc.pb.cc src/arrow/flight/Flight.grpc.pb.h /tmp/arrow-build/src/arrow/flight/Flight.pb.cc /tmp/arrow-build/src/arrow/flight/Flight.pb.h /tmp/arrow-build/src/arrow/flight/Flight.grpc.pb.cc /tmp/arrow-build/src/arrow/flight/Flight.grpc.pb.h 
cd /tmp/arrow-build/src/arrow/flight && /opt/vcpkg/installed/x64-linux/tools/protobuf/protoc -I/arrow/cpp/../format --cpp_out=/tmp/arrow-build/src/arrow/flight /arrow/cpp/../format/Flight.proto && /opt/vcpkg/installed/x64-linux/tools/protobuf/protoc -I/arrow/cpp/../format --grpc_out=/tmp/arrow-build/src/arrow/flight --plugin=protoc-gen-grpc=/opt/vcpkg/installed/amd64-linux-static-release/../x64-linux/tools/grpc/grpc_cpp_plugin /arrow/cpp/../format/Flight.proto
--grpc_out: protoc-gen-grpc: Plugin killed by signal 11.

raulcd · 2024-07-26T15:26:03Z

@github-actions crossbow submit wheel-macos-big-sur-cp310-arm64

github-actions · 2024-07-26T15:28:27Z

Revision: 2cedbe6ece02255aa5564bd285d5e2474bef1fdf

Submitted crossbow builds: ursacomputing/crossbow @ actions-08db436a21

Task	Status
wheel-macos-big-sur-cp310-arm64

kou · 2024-07-29T00:58:41Z

Hmm. It seems that https://github.com/apache/orc/blob/23044d79a26cf8792bcb1936ab601308b0781962/cmake_modules/ThirdpartyToolchain.cmake#L416-L418 wasn't used. It means that Protobuf_ROOT is an empty string in this case:

arrow/cpp/cmake_modules/ThirdpartyToolchain.cmake

Lines 4524 to 4529 in 187197c

    
               get_target_property(PROTOBUF_INCLUDE_DIR ${ARROW_PROTOBUF_LIBPROTOBUF} 
        
                                   INTERFACE_INCLUDE_DIRECTORIES) 
        
               get_filename_component(Protobuf_ROOT "${PROTOBUF_INCLUDE_DIR}" DIRECTORY) 
        
               set(PROTOBUF_HOME 
        
                   ${Protobuf_ROOT} 
        
                   CACHE STRING "" FORCE)

Could you debug print Protobuf_ROOT value?

BTW, the ORC code should be fixed:

diff --git a/cmake_modules/ThirdpartyToolchain.cmake b/cmake_modules/ThirdpartyToolchain.cmake
index 79324984a..f0808715b 100644
--- a/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cmake_modules/ThirdpartyToolchain.cmake
@@ -466,13 +466,13 @@ if (ORC_PACKAGE_KIND STREQUAL "conan")
 elseif (NOT "${PROTOBUF_HOME}" STREQUAL "")
   find_package (Protobuf REQUIRED)
 
-  if (ORC_PREFER_STATIC_PROTOBUF AND ${PROTOBUF_STATIC_LIB})
+  if (ORC_PREFER_STATIC_PROTOBUF AND PROTOBUF_STATIC_LIB)
     add_resolved_library (orc_protobuf ${PROTOBUF_STATIC_LIB} ${PROTOBUF_INCLUDE_DIR})
   else ()
     add_resolved_library (orc_protobuf ${PROTOBUF_LIBRARY} ${PROTOBUF_INCLUDE_DIR})
   endif ()
 
-  if (ORC_PREFER_STATIC_PROTOBUF AND ${PROTOC_STATIC_LIB})
+  if (ORC_PREFER_STATIC_PROTOBUF AND PROTOC_STATIC_LIB)
     add_resolved_library (orc_protoc ${PROTOC_STATIC_LIB} ${PROTOBUF_INCLUDE_DIR})
   else ()
     add_resolved_library (orc_protoc ${PROTOC_LIBRARY} ${PROTOBUF_INCLUDE_DIR})

@wgtmac Could you fix the ORC side problem (that is not related to this PR's failure)? We should not use ${...} in these if (...)s because it may be "not defined" or "an empty string". In these case, the if (...) becomes if (A AND). It's a syntax error. ORC has similar problem for other libraries such as LZ4: https://github.com/apache/orc/blob/37201cb8a186d2111a5ccf49b213003c4c947859/cmake_modules/ThirdpartyToolchain.cmake#L313

wgtmac · 2024-07-29T04:56:58Z

@kou Sorry for the issue caused by ORC. Let me take a look.

kou · 2024-07-29T06:45:10Z

Thanks!

raulcd · 2024-07-29T08:59:10Z

Hmm. It seems that https://github.com/apache/orc/blob/23044d79a26cf8792bcb1936ab601308b0781962/cmake_modules/ThirdpartyToolchain.cmake#L416-L418 wasn't used. It means that Protobuf_ROOT is an empty string in this case:

arrow/cpp/cmake_modules/ThirdpartyToolchain.cmake

Lines 4524 to 4529 in 187197c

get_target_property(PROTOBUF_INCLUDE_DIR ${ARROW_PROTOBUF_LIBPROTOBUF}

INTERFACE_INCLUDE_DIRECTORIES)

get_filename_component(Protobuf_ROOT "${PROTOBUF_INCLUDE_DIR}" DIRECTORY)

set(PROTOBUF_HOME

${Protobuf_ROOT}

CACHE STRING "" FORCE)

Could you debug print Protobuf_ROOT value?

-- Building Apache ORC from source
-- ## Protobuf_ROOT: /opt/vcpkg/installed/amd64-linux-static-release
-- Build type: RELEASE

@kou you lost me with this comment, are you talking about the ORC issue here or the flight issue with protobuf?

other logs about where protoc is:

-- Found Protobuf: /opt/vcpkg/installed/x64-linux/tools/protobuf/protoc (found version "25.1.0")
-- Providing CMake module for FindProtobufAlt as part of Arrow CMake package
-- Found protoc: /opt/vcpkg/installed/x64-linux/tools/protobuf/protoc

kou · 2024-07-29T09:44:15Z

you lost me with this comment, are you talking about the ORC issue here or the flight issue with protobuf?

The ORC issue here. I don't think that the cpp/cmake_modules/orc.diff is the right approach.
Sorry. I haven't looked at #43417 (comment) yet.

raulcd · 2024-07-29T09:50:54Z

I don't think that the cpp/cmake_modules/orc.diff is the right approach.

ah! ok, that makes sense. Thanks for your help!

kou · 2024-07-30T08:02:00Z

(I hope that I can take a look at the Flight issue tomorrow...)

raulcd · 2024-07-30T08:17:32Z

(I hope that I can take a look at the Flight issue tomorrow...)

Thanks @kou, no worries, I am testing the orc issue and it seems like the FindProtobuf file from ORC isn't used. I'll keep testing and will share findings.

raulcd · 2024-07-30T09:27:23Z

ok, vcpkg uses CONFIG for find_package and does not use FindProtobuf.cmake but ProtobufConfig.cmake or protobuf-config.cmake based on debug:

    /opt/vcpkg/installed/amd64-linux-static-release/share/protobuf/ProtobufConfig.cmake
    /opt/vcpkg/installed/amd64-linux-static-release/share/protobuf/protobuf-config.cmake

This does not populate PROTOBUF_STATIC_LIB nor PROTOC_STATIC_LIB, the following snippet for orc.diff fixes the issue:

diff --git a/cmake_modules/ThirdpartyToolchain.cmake b/cmake_modules/ThirdpartyToolchain.cmake
index aa520ff..7bd75d1 100644
--- a/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cmake_modules/ThirdpartyToolchain.cmake
@@ -416,6 +416,18 @@ endif ()
 if (NOT "${PROTOBUF_HOME}" STREQUAL "" OR ORC_PACKAGE_KIND STREQUAL "conan")
   find_package (Protobuf REQUIRED)
   set(PROTOBUF_VENDORED FALSE)
+  if (Protobuf_FOUND AND NOT PROTOBUF_STATIC_LIB)
+    get_target_property (target_type protobuf::libprotobuf TYPE)
+    if (target_type STREQUAL "STATIC_LIBRARY")
+        set(PROTOBUF_STATIC_LIB protobuf::libprotobuf)
+    endif ()
+  endif()
+  if (Protobuf_FOUND AND NOT PROTOC_STATIC_LIB)
+    get_target_property (target_type protobuf::libprotoc TYPE)
+    if (target_type STREQUAL "STATIC_LIBRARY")
+        set (PROTOC_STATIC_LIB protobuf::libprotoc)
+    endif ()
+  endif()
 else ()
   set(PROTOBUF_PREFIX "${THIRDPARTY_DIR}/protobuf_ep-install")
   set(PROTOBUF_INCLUDE_DIR "${PROTOBUF_PREFIX}/include")

raulcd · 2024-07-30T10:00:18Z

I've tried to debug the protoc issue but I've had no luck, adding it here in case it can help:

[root@17eac57d5aef flight]# gdb --args /opt/vcpkg/installed/x64-linux/tools/protobuf/protoc -I/arrow/cpp/../format --grpc_out=/tmp/arrow-build/src/arrow/flight --plugin=protoc-gen-grpc=/opt/vcpkg/installed/amd64-linux-static-release/../x64-linux/tools/grpc/grpc_cpp_plugin /arrow/cpp/../format/Flight.proto
...
(gdb) set follow-fork-mode child
(gdb) run
Starting program: /opt/vcpkg/installed/x64-linux/tools/protobuf/protoc -I/arrow/cpp/../format --grpc_out=/tmp/arrow-build/src/arrow/flight --plugin=protoc-gen-grpc=/opt/vcpkg/installed/amd64-linux-static-release/../x64-linux/tools/grpc/grpc_cpp_plugin /arrow/cpp/../format/Flight.proto
warning: Error disabling address space randomization: Operation not permitted
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[Attaching after Thread 0x799b682acc00 (LWP 642) fork to child process 643]
[New inferior 2 (process 643)]
[Detaching after fork from parent process 642]
[Inferior 1 (process 642) detached]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
process 643 is executing new program: /opt/vcpkg/installed/x64-linux/tools/grpc/grpc_cpp_plugin

Thread 2.1 "grpc_cpp_plugin" received signal SIGSEGV, Segmentation fault.
[Switching to process 643]
0x00007e16ee5638a4 in dl_main (phdr=<optimized out>, phnum=<optimized out>, user_entry=<optimized out>, auxv=<optimized out>) at rtld.c:1832
1832	rtld.c: No such file or directory.

I tried setting the following to docker-compose.yml as suggested on stackoverflow on the Error disabling address space randomization: Operation not permitted to try and remove that warning without luck (I don't think is related, though):

diff --git a/docker-compose.yml b/docker-compose.yml
index cf22324..674fe0a 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -1132,6 +1132,10 @@ services:
       - .:/arrow:delegated
       - ${DOCKER_VOLUME_PREFIX}python-wheel-manylinux-2-28-ccache:/ccache:delegated
     command: /arrow/ci/scripts/python_wheel_manylinux_build.sh
+    cap_add:
+      - SYS_PTRACE
+    security_opt:
+      - apparmor:unconfined
 
   python-wheel-manylinux-test-imports:
     image: ${ARCH}/python:${PYTHON}

It seems that just executing grpc_cpp_plugin segfaults:

[root@849e8f917b46 flight]# /opt/vcpkg/installed/x64-linux/tools/grpc/grpc_cpp_plugin
Segmentation fault (core dumped)

edit: added some more info.

kou · 2024-07-31T02:23:26Z

This problem #43417 (comment) may be already solved in upstream: apache/orc#1963
(But not released yet.)

dongjoon-hyun · 2024-07-31T15:30:57Z

FYI, Apache ORC 2.0.2 will be scheduled in 2 weeks with the following. Is that the only one Apache ORC community fixes?

ORC-1732: [C++] Fix detecting Homebrew-installed Protobuf on MacOS orc#1963

kou · 2024-08-01T01:15:37Z

Thanks for sharing the information.
We haven't reported #43417 (comment) to Apache ORC yet but it's nice to have in Apache ORC 2.0.2. But it's not a blocker of this PR. So it's not in a hurry.

wgtmac · 2024-08-01T02:05:27Z

@luffy-zh has opened https://issues.apache.org/jira/browse/ORC-1751 and will fix it in 2.0.2.

kou · 2024-08-01T02:08:20Z

Wow! Thanks!

What changes were proposed in this pull request? fix syntax error in ThirdpartyToolchain Why are the changes needed? Handle the issue discussed [here]( apache/arrow#43417) How was this patch tested? Test it locally Was this patch authored or co-authored using generative AI tooling? NO Closes #1994 from luffy-zh/ORC-1751. Authored-by: luffy-zh <zhnice@outlook.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

### What changes were proposed in this pull request? fix syntax error in ThirdpartyToolchain ### Why are the changes needed? Handle the issue discussed [here]( apache/arrow#43417) ### How was this patch tested? Test it locally ### Was this patch authored or co-authored using generative AI tooling? NO Closes #1997 from luffy-zh/branch-2.0. Authored-by: luffy-zh <zhnice@outlook.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

raulcd · 2024-08-27T10:09:22Z

@github-actions crossbow submit wheel-manylinux-2-28-cp39-amd64

github-actions · 2024-08-27T10:11:38Z

Revision: e939b3d

Submitted crossbow builds: ursacomputing/crossbow @ actions-40f30280ce

Task	Status
wheel-manylinux-2-28-cp39-amd64

pitrou · 2024-11-28T16:47:34Z

@raulcd What is the status on this?

raulcd · 2024-11-28T17:04:03Z

This one is mainly abandoned. I should probably close this one and try with a newer vcpkg version even though some of the existing changes here might still apply.

raulcd · 2024-12-03T14:55:39Z

Closing this old one. Trying again here: #44919

### What changes were proposed in this pull request? Add an CMake option ORC_FORMAT_URL to indicate where to download the orc-format_ep ### Why are the changes needed? Handle the issue discussed apache/arrow#43417 ### How was this patch tested? Test it locally ### Was this patch authored or co-authored using generative AI tooling? NO Closes #2094 from luffy-zh/ORC-1810. Authored-by: luffy-zh <zhnice@outlook.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

github-actions bot added the awaiting committer review Awaiting committer review label Jul 25, 2024

raulcd mentioned this pull request Jul 25, 2024

[C++] Compiling with vcpkg dependencies is broken for latest VS 2022 (17.10) #43327

Open

This comment was marked as outdated.

Sign in to view

github-actions bot added the Component: C++ label Jul 26, 2024

raulcd force-pushed the GH-43416 branch 2 times, most recently from 2abb87d to ee89232 Compare July 26, 2024 10:18

raulcd force-pushed the GH-43416 branch from daf593c to ccb7ee7 Compare August 2, 2024 09:52

luffy-zh mentioned this pull request Aug 5, 2024

ORC-1751: [C++] fix syntax error in ThirdpartyToolchain apache/orc#1994

Closed

luffy-zh mentioned this pull request Aug 7, 2024

ORC-1751: [C++] fix syntax error in ThirdpartyToolchain apache/orc#1997

Closed

raulcd added 7 commits August 27, 2024 11:09

apacheGH-43416: [CI] Upgrade our vcpkg version on .env

0c59106

Fix patches

e20f903

Apply orc patch in order to make CMake happy

8880226

Add license to patch file

dce3329

Make Windows happy with the patch

4d155c8

Update patch

a0e017f

New snippet to fix orc

e939b3d

raulcd force-pushed the GH-43416 branch from ccb7ee7 to e939b3d Compare August 27, 2024 09:11

kou mentioned this pull request Aug 27, 2024

[C++] Update vcpkg to use boost-cmake #43812

Closed

raulcd closed this Dec 3, 2024

luffy-zh mentioned this pull request Dec 24, 2024

ORC-1810: [C++] Add environment variable ORC_FORMAT_URL apache/orc#2094

Closed

Conversation

raulcd commented Jul 25, 2024 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions bot commented Jul 25, 2024

Uh oh!

raulcd commented Jul 25, 2024

Uh oh!

github-actions bot commented Jul 25, 2024

Uh oh!

raulcd commented Jul 25, 2024

Uh oh!

This comment was marked as outdated.

raulcd commented Jul 25, 2024

Uh oh!

github-actions bot commented Jul 25, 2024

Uh oh!

raulcd commented Jul 25, 2024

Uh oh!

github-actions bot commented Jul 25, 2024

Uh oh!

raulcd commented Jul 25, 2024

Uh oh!

raulcd commented Jul 26, 2024

Uh oh!

raulcd commented Jul 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jul 26, 2024

Uh oh!

raulcd commented Jul 26, 2024

Uh oh!

raulcd commented Jul 26, 2024

Uh oh!

github-actions bot commented Jul 26, 2024

Uh oh!

kou commented Jul 29, 2024

Uh oh!

wgtmac commented Jul 29, 2024

Uh oh!

kou commented Jul 29, 2024

Uh oh!

raulcd commented Jul 29, 2024

Uh oh!

kou commented Jul 29, 2024

Uh oh!

raulcd commented Jul 29, 2024

Uh oh!

kou commented Jul 30, 2024

Uh oh!

raulcd commented Jul 30, 2024

Uh oh!

raulcd commented Jul 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

raulcd commented Jul 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kou commented Jul 31, 2024

Uh oh!

dongjoon-hyun commented Jul 31, 2024 • edited by kou Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kou commented Aug 1, 2024

Uh oh!

wgtmac commented Aug 1, 2024

Uh oh!

kou commented Aug 1, 2024

Uh oh!

raulcd commented Aug 27, 2024

Uh oh!

github-actions bot commented Aug 27, 2024

Uh oh!

pitrou commented Nov 28, 2024

Uh oh!

raulcd commented Nov 28, 2024

raulcd commented Jul 25, 2024 •

edited by github-actions bot

Loading

raulcd commented Jul 26, 2024 •

edited

Loading

raulcd commented Jul 30, 2024 •

edited

Loading

raulcd commented Jul 30, 2024 •

edited

Loading

dongjoon-hyun commented Jul 31, 2024 •

edited by kou

Loading