Skip to content

[thirdparty] (buildscript) use PKG_CONFIG_LIBDIR instead of PKG_CONFIG_PATH#60752

Open
SwingURM wants to merge 1 commit intoapache:masterfrom
SwingURM:thirdparty/fix-pkg-config
Open

[thirdparty] (buildscript) use PKG_CONFIG_LIBDIR instead of PKG_CONFIG_PATH#60752
SwingURM wants to merge 1 commit intoapache:masterfrom
SwingURM:thirdparty/fix-pkg-config

Conversation

@SwingURM
Copy link

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
When compiling hyperscan, the container's behavior is to give up compiling hscollider because it cannot find pcre.

[ ~]# pkg-config --exists libpcre && echo "success"
[ ~]# env | grep PKG
PKG_CONFIG_PATH=/root/doris/thirdparty/installed/lib64/pkgconfig

On my local machine, PKG_CONFIG_PATH (which is set in the script) does not prevent pkg-config from searching system libs. Thus, CMake's pkg_check_modules finds pcre and tries to compile hscollider, but fails due to mixing system's libc (linker errors).

~/ldb_toolchain/bin via  v3.14.2 
❯ ./pkg-config --exists libpcre && echo "find pcre"
find pcre

~/ldb_toolchain/bin via  v3.14.2 
❯ env | grep PKG                                   
PKG_CONFIG_PATH=/home/iamswing/doris/thirdparty/installed/lib64/pkgconfig

However, on my local machine,
Error:

src/hyperscan-5.4.2/doris_build on  master [!?] via  v15.2.1-gcc via  v4.2.3
❯ /home/iamswing/ldb_toolchain/bin/clang++ -D_HAS_AUTO_PTR_ETC=0 -O2 -std=c++11 -Wall -Wextra -Wshadow -Wswitch -Wreturn-type -Wcast-qual -Wno-deprecated -Wnon-virtual-dtor -fno-strict-aliasing -DNDEBUG -fno-omit-frame-pointer -fvisibility=hidden -Wvla -Wpointer-arith -Wno-self-assign -Wno-unused-const-variable -Wno-unused-but-set-variable -Wno-bitwise-instead-of-logical -Wno-ignored-attributes -Wno-redundant-move -Wweak-vtables -Wmissing-declarations  -g -DNDEBUG  tools/hscollider/CMakeFiles/hscollider.dir/Corpora.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/FileCorpora.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/ColliderCorporaParser.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/NfaGeneratedCorpora.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/GraphTruth.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/GroundTruth.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/UltimateTruth.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/args.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/limit.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/pcre_util.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/sig.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/Thread.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/main.cpp.o -o bin/hscollider  lib/libhs.a  -L/usr/lib  -lpcre  lib/libdatabaseutil.a  lib/libexpressionutil.a  lib/libcorpusomatic.a  lib/libcrosscompileutil.a  -lpthread  -rdynamic -v
clang version 20.1.7
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /home/iamswing/ldb_toolchain/tmp/gentoo/llvm/bin
System configuration file directory: /home/iamswing/ldb_toolchain/bin/../etc/clang
User configuration file directory: /home/iamswing/.config/clang
Selected GCC installation: /home/iamswing/ldb_toolchain/bin/../lib/gcc/x86_64-pc-linux-gnu/15
Candidate multilib: .;@m64
Selected multilib: .;@m64
 "/home/iamswing/ldb_toolchain/bin/x86_64-pc-linux-gnu-ld" --sysroot=/home/iamswing/ldb_toolchain/bin/.. --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -export-dynamic -pie -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o bin/hscollider /home/iamswing/ldb_toolchain/bin/../usr/lib/Scrt1.o /home/iamswing/ldb_toolchain/bin/../usr/lib/crti.o /home/iamswing/ldb_toolchain/bin/../lib/gcc/x86_64-pc-linux-gnu/15/crtbeginS.o -L/usr/lib -L/home/iamswing/ldb_toolchain/bin/../lib/gcc/x86_64-pc-linux-gnu/15 -L/home/iamswing/ldb_toolchain/bin/../lib -L/home/iamswing/ldb_toolchain/bin/../usr/lib tools/hscollider/CMakeFiles/hscollider.dir/Corpora.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/FileCorpora.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/ColliderCorporaParser.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/NfaGeneratedCorpora.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/GraphTruth.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/GroundTruth.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/UltimateTruth.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/args.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/limit.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/pcre_util.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/sig.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/Thread.cpp.o tools/hscollider/CMakeFiles/hscollider.dir/main.cpp.o lib/libhs.a -lpcre lib/libdatabaseutil.a lib/libexpressionutil.a lib/libcorpusomatic.a lib/libcrosscompileutil.a -lpthread -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /home/iamswing/ldb_toolchain/bin/../lib/gcc/x86_64-pc-linux-gnu/15/crtendS.o /home/iamswing/ldb_toolchain/bin/../usr/lib/crtn.o
x86_64-pc-linux-gnu-ld: error: undefined symbol: __libc_csu_fini
>>> referenced by /home/iamswing/ldb_toolchain/bin/../usr/lib/Scrt1.o:(_start)

x86_64-pc-linux-gnu-ld: error: undefined symbol: __libc_csu_init
>>> referenced by /home/iamswing/ldb_toolchain/bin/../usr/lib/Scrt1.o:(_start)
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

Solution

Use PKG_CONFIG_LIBDIR instead of PKG_CONFIG_PATH. PKG_CONFIG_PATH appends to the default search path, causing CMake to find the system's pcre. Using PKG_CONFIG_LIBDIR overrides the default search path and only searches in the specified directory. All relevant .pc files are present in this path.

Modifying CMake's related variables may also solve this problem.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • [☑️] Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • [☑️] No.
    • Yes.
  • Does this need documentation?

    • [☑️] No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Feb 13, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@SwingURM SwingURM changed the title [thirdparty] use PKG_CONFIG_LIBDIR instead of PKG_CONFIG_PATH [thirdparty] (buildscript) use PKG_CONFIG_LIBDIR instead of PKG_CONFIG_PATH Feb 13, 2026
@SwingURM
Copy link
Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 28471 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4f8196b36fd432fc1e6911e7f3bf764f4a4abcd8, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17652	4467	4301	4301
q2	q3	10654	768	515	515
q4	4686	351	249	249
q5	7556	1210	1017	1017
q6	171	174	144	144
q7	766	825	671	671
q8	9294	1457	1311	1311
q9	5123	4751	4682	4682
q10	6840	1867	1624	1624
q11	486	263	253	253
q12	695	561	467	467
q13	17762	4215	3389	3389
q14	231	231	215	215
q15	963	789	799	789
q16	747	709	659	659
q17	709	857	438	438
q18	6021	5423	5143	5143
q19	1163	993	602	602
q20	515	496	374	374
q21	4479	1827	1389	1389
q22	343	288	239	239
Total cold run time: 96856 ms
Total hot run time: 28471 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4397	4348	4342	4342
q2	q3	1773	2165	1706	1706
q4	824	1144	813	813
q5	4033	4318	4321	4318
q6	174	179	142	142
q7	1703	1602	1492	1492
q8	2406	2653	2491	2491
q9	7445	7352	7357	7352
q10	2713	2825	2455	2455
q11	515	431	426	426
q12	493	581	434	434
q13	3990	4605	3691	3691
q14	277	310	290	290
q15	868	818	846	818
q16	771	836	799	799
q17	1196	1540	1297	1297
q18	7063	6813	6756	6756
q19	894	909	841	841
q20	2065	2163	2127	2127
q21	3959	3472	3277	3277
q22	457	448	412	412
Total cold run time: 48016 ms
Total hot run time: 46279 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184583 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4f8196b36fd432fc1e6911e7f3bf764f4a4abcd8, data reload: false

query5	5029	658	537	537
query6	319	219	198	198
query7	4221	487	265	265
query8	346	244	234	234
query9	8694	2765	2791	2765
query10	541	388	353	353
query11	16931	17408	17313	17313
query12	230	128	130	128
query13	1380	481	351	351
query14	7335	3446	3243	3243
query14_1	2952	3015	2986	2986
query15	215	202	182	182
query16	1013	491	478	478
query17	1245	816	673	673
query18	2854	487	357	357
query19	223	229	206	206
query20	151	133	144	133
query21	213	149	123	123
query22	5438	5102	4837	4837
query23	17231	16824	16588	16588
query23_1	16603	16707	16681	16681
query24	7171	1705	1271	1271
query24_1	1261	1260	1274	1260
query25	550	485	443	443
query26	1234	273	159	159
query27	2748	489	289	289
query28	4443	1875	1949	1875
query29	794	595	493	493
query30	309	241	218	218
query31	877	717	636	636
query32	86	73	76	73
query33	518	345	285	285
query34	939	944	576	576
query35	644	679	585	585
query36	1078	1082	926	926
query37	132	98	86	86
query38	2955	2913	2915	2913
query39	884	894	849	849
query39_1	834	834	831	831
query40	224	151	133	133
query41	67	58	56	56
query42	106	103	104	103
query43	376	397	357	357
query44	
query45	197	192	190	190
query46	938	1028	628	628
query47	2137	2133	2135	2133
query48	313	334	222	222
query49	636	472	384	384
query50	716	282	220	220
query51	4096	4170	4077	4077
query52	107	113	101	101
query53	337	368	305	305
query54	297	269	270	269
query55	89	84	79	79
query56	313	307	294	294
query57	1338	1331	1247	1247
query58	294	273	272	272
query59	2525	2645	2619	2619
query60	355	333	316	316
query61	143	146	142	142
query62	624	590	550	550
query63	330	287	286	286
query64	4861	1266	987	987
query65	
query66	1381	496	359	359
query67	16343	16468	16333	16333
query68	
query69	398	307	294	294
query70	990	991	861	861
query71	337	316	299	299
query72	2797	2698	2434	2434
query73	550	588	327	327
query74	9938	9941	9757	9757
query75	2996	2854	2526	2526
query76	2297	1128	706	706
query77	380	387	312	312
query78	11201	11453	10663	10663
query79	2383	856	612	612
query80	1772	662	576	576
query81	550	285	249	249
query82	997	160	124	124
query83	345	270	244	244
query84	248	124	103	103
query85	912	482	430	430
query86	420	310	288	288
query87	3218	3092	2962	2962
query88	3614	2683	2671	2671
query89	451	381	366	366
query90	1955	185	172	172
query91	159	158	131	131
query92	74	74	77	74
query93	1046	888	507	507
query94	624	326	290	290
query95	581	340	376	340
query96	654	550	234	234
query97	2478	2491	2437	2437
query98	220	209	220	209
query99	995	971	891	891
Total cold run time: 256564 ms
Total hot run time: 184583 ms

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.69% (19523/37050)
Line Coverage 36.25% (182046/502248)
Region Coverage 32.62% (141273/433120)
Branch Coverage 33.64% (61220/181963)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.80% (26065/36303)
Line Coverage 54.49% (272998/500999)
Region Coverage 52.03% (227640/437497)
Branch Coverage 53.46% (97659/182667)

@SwingURM
Copy link
Author

run p0

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.80% (26064/36303)
Line Coverage 54.48% (272962/500999)
Region Coverage 52.02% (227567/437497)
Branch Coverage 53.44% (97621/182667)

@doris-robot
Copy link

TPC-H: Total hot run time: 28722 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4f8196b36fd432fc1e6911e7f3bf764f4a4abcd8, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17619	4434	4294	4294
q2	q3	10654	763	523	523
q4	4680	351	252	252
q5	7552	1199	1023	1023
q6	177	171	144	144
q7	758	831	666	666
q8	9530	1382	1294	1294
q9	4878	4742	4705	4705
q10	6798	1861	1620	1620
q11	446	257	243	243
q12	737	566	478	478
q13	17776	4206	3367	3367
q14	221	227	236	227
q15	946	829	793	793
q16	754	717	673	673
q17	720	885	443	443
q18	6024	5315	5259	5259
q19	1231	988	629	629
q20	523	515	385	385
q21	4541	1894	1451	1451
q22	347	289	253	253
Total cold run time: 96912 ms
Total hot run time: 28722 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4416	4363	4331	4331
q2	q3	1750	2163	1711	1711
q4	845	1156	756	756
q5	4015	4298	4286	4286
q6	179	173	142	142
q7	1716	1598	1482	1482
q8	2437	2629	2536	2536
q9	7350	7385	7359	7359
q10	2684	2901	2396	2396
q11	512	456	418	418
q12	489	581	450	450
q13	3999	4567	3593	3593
q14	276	299	272	272
q15	840	799	803	799
q16	733	778	739	739
q17	1199	1564	1353	1353
q18	7127	6820	6609	6609
q19	986	959	884	884
q20	2069	2126	2015	2015
q21	4113	3461	3300	3300
q22	472	457	394	394
Total cold run time: 48207 ms
Total hot run time: 45825 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183381 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4f8196b36fd432fc1e6911e7f3bf764f4a4abcd8, data reload: false

query5	4838	639	520	520
query6	327	226	238	226
query7	4229	476	281	281
query8	334	237	232	232
query9	8754	2751	2776	2751
query10	530	403	327	327
query11	16891	17667	17222	17222
query12	277	134	125	125
query13	1406	475	350	350
query14	7904	3298	3018	3018
query14_1	2974	2990	2980	2980
query15	210	208	194	194
query16	1009	398	489	398
query17	1435	772	671	671
query18	2800	476	367	367
query19	225	216	192	192
query20	154	143	142	142
query21	234	149	133	133
query22	5450	5033	4905	4905
query23	17161	16764	16517	16517
query23_1	16628	16622	16802	16622
query24	6858	1626	1223	1223
query24_1	1228	1220	1225	1220
query25	559	493	429	429
query26	1246	266	152	152
query27	2740	471	333	333
query28	4419	1831	1871	1831
query29	811	548	460	460
query30	312	245	203	203
query31	886	728	645	645
query32	78	67	65	65
query33	510	318	272	272
query34	898	890	565	565
query35	650	662	598	598
query36	1058	1127	953	953
query37	145	98	86	86
query38	2952	2905	2880	2880
query39	893	866	844	844
query39_1	836	807	839	807
query40	234	148	136	136
query41	61	59	58	58
query42	102	100	101	100
query43	374	384	355	355
query44	
query45	198	192	186	186
query46	892	975	604	604
query47	2135	2138	2078	2078
query48	307	317	233	233
query49	624	463	370	370
query50	666	274	212	212
query51	4090	4098	4186	4098
query52	109	104	98	98
query53	292	337	273	273
query54	287	257	249	249
query55	90	81	81	81
query56	352	322	311	311
query57	1376	1324	1284	1284
query58	290	277	264	264
query59	2582	2672	2566	2566
query60	326	339	325	325
query61	153	143	148	143
query62	622	580	556	556
query63	307	280	273	273
query64	4864	1263	991	991
query65	
query66	1371	450	355	355
query67	16169	16225	16269	16225
query68	
query69	398	299	279	279
query70	968	973	949	949
query71	326	305	293	293
query72	2736	2671	2390	2390
query73	532	543	316	316
query74	9921	9911	9711	9711
query75	2820	2730	2444	2444
query76	2296	1051	673	673
query77	374	371	312	312
query78	11120	11337	10721	10721
query79	3153	813	588	588
query80	1797	612	515	515
query81	586	269	247	247
query82	1003	142	112	112
query83	337	258	238	238
query84	256	115	100	100
query85	875	474	431	431
query86	502	308	313	308
query87	3106	3107	2996	2996
query88	3605	2664	2647	2647
query89	438	375	335	335
query90	2051	171	168	168
query91	165	156	131	131
query92	85	69	70	69
query93	1492	811	499	499
query94	642	324	288	288
query95	579	342	302	302
query96	632	514	229	229
query97	2457	2472	2418	2418
query98	233	221	219	219
query99	1010	970	910	910
Total cold run time: 257531 ms
Total hot run time: 183381 ms

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.67% (19533/37083)
Line Coverage 36.23% (182084/502562)
Region Coverage 32.58% (141293/433615)
Branch Coverage 33.61% (61232/182166)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.67% (26041/36336)
Line Coverage 54.35% (272459/501313)
Region Coverage 51.82% (226977/437992)
Branch Coverage 53.26% (97398/182870)

1 similar comment
@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.67% (26041/36336)
Line Coverage 54.35% (272459/501313)
Region Coverage 51.82% (226977/437992)
Branch Coverage 53.26% (97398/182870)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.66% (26039/36336)
Line Coverage 54.35% (272465/501313)
Region Coverage 51.79% (226847/437992)
Branch Coverage 53.26% (97392/182870)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

Comments