Skip to content

[bugfix](topn) add schema to merge iterator#60774

Open
yiguolei wants to merge 11 commits intoapache:masterfrom
yiguolei:f2-output-schema
Open

[bugfix](topn) add schema to merge iterator#60774
yiguolei wants to merge 11 commits intoapache:masterfrom
yiguolei:f2-output-schema

Conversation

@yiguolei
Copy link
Contributor

@yiguolei yiguolei commented Feb 15, 2026

What problem does this PR solve?

#60772 This PR want to fix topn bug, In merge iterator, should always use input schema to generate block. But schema change process will generate a ref block contains delete predicate in #57191. So the code is very complex, So that in this PR, I simplify all the logic:

  1. add a input schema to merge iterator.
  2. using input schema in all merge iterator and union iterator.
  3. change schema change logic not use delete predicate column.

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@yiguolei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 28655 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1cd5e08ba967f96af984daf751d5835666f973b4, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17596	4464	4287	4287
q2	q3	10665	774	509	509
q4	4678	352	247	247
q5	7539	1204	1023	1023
q6	169	174	147	147
q7	802	838	663	663
q8	9298	1440	1305	1305
q9	4833	4706	4708	4706
q10	6805	1892	1609	1609
q11	458	268	238	238
q12	714	567	477	477
q13	17807	4220	3407	3407
q14	225	235	224	224
q15	967	787	786	786
q16	745	722	663	663
q17	709	872	419	419
q18	5903	5474	5240	5240
q19	1312	997	632	632
q20	556	508	408	408
q21	4753	1886	1417	1417
q22	342	298	248	248
Total cold run time: 96876 ms
Total hot run time: 28655 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4446	4410	4401	4401
q2	q3	1794	2171	1716	1716
q4	869	1180	772	772
q5	4014	4375	4301	4301
q6	180	180	143	143
q7	1740	1595	1506	1506
q8	2441	2668	2545	2545
q9	7489	7455	7431	7431
q10	2739	2918	2450	2450
q11	512	447	434	434
q12	509	580	459	459
q13	4013	4421	3585	3585
q14	273	291	274	274
q15	825	790	804	790
q16	718	738	736	736
q17	1144	1563	1349	1349
q18	7192	6577	6654	6577
q19	956	976	970	970
q20	2103	2148	1973	1973
q21	3907	3687	3330	3330
q22	484	450	399	399
Total cold run time: 48348 ms
Total hot run time: 46141 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 182999 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1cd5e08ba967f96af984daf751d5835666f973b4, data reload: false

query5	4950	632	482	482
query6	335	226	210	210
query7	4213	463	262	262
query8	339	247	238	238
query9	8769	2752	2705	2705
query10	542	391	325	325
query11	16713	17165	17073	17073
query12	207	131	128	128
query13	1541	468	351	351
query14	6958	3246	2989	2989
query14_1	2904	2909	2880	2880
query15	222	212	193	193
query16	1053	494	462	462
query17	1332	726	660	660
query18	3049	442	346	346
query19	214	220	183	183
query20	142	131	127	127
query21	217	141	131	131
query22	4918	5117	4909	4909
query23	17392	16963	16815	16815
query23_1	16826	16921	16924	16921
query24	7072	1604	1208	1208
query24_1	1236	1224	1253	1224
query25	558	462	404	404
query26	1236	259	152	152
query27	2776	465	284	284
query28	4440	1859	1850	1850
query29	809	555	461	461
query30	310	242	208	208
query31	876	720	639	639
query32	79	70	71	70
query33	511	326	282	282
query34	907	905	571	571
query35	667	674	606	606
query36	1074	1118	1009	1009
query37	131	92	81	81
query38	2903	2831	2838	2831
query39	855	854	845	845
query39_1	823	816	805	805
query40	225	148	135	135
query41	65	63	60	60
query42	105	101	102	101
query43	369	374	357	357
query44	
query45	205	189	184	184
query46	877	982	603	603
query47	2136	2146	2086	2086
query48	305	323	231	231
query49	623	466	379	379
query50	685	273	211	211
query51	4097	4127	4036	4036
query52	103	107	95	95
query53	290	335	281	281
query54	291	264	250	250
query55	90	79	84	79
query56	308	317	309	309
query57	1353	1344	1266	1266
query58	292	276	268	268
query59	2734	2686	2564	2564
query60	327	330	320	320
query61	149	145	149	145
query62	642	603	534	534
query63	312	268	268	268
query64	4897	1362	1092	1092
query65	
query66	1377	473	361	361
query67	16088	16148	15991	15991
query68	
query69	440	302	284	284
query70	950	985	961	961
query71	331	311	292	292
query72	2742	2632	2366	2366
query73	538	535	315	315
query74	9942	9882	9688	9688
query75	2814	2753	2454	2454
query76	2299	1029	674	674
query77	359	387	304	304
query78	11174	11327	10675	10675
query79	2546	791	608	608
query80	1830	610	533	533
query81	561	287	256	256
query82	1033	149	115	115
query83	325	263	238	238
query84	253	114	98	98
query85	899	477	438	438
query86	427	309	299	299
query87	3082	3064	3012	3012
query88	3503	2710	2669	2669
query89	425	365	345	345
query90	2001	170	173	170
query91	162	158	132	132
query92	75	74	67	67
query93	1057	811	507	507
query94	646	316	299	299
query95	564	394	308	308
query96	637	549	229	229
query97	2501	2511	2452	2452
query98	224	217	215	215
query99	963	1007	912	912
Total cold run time: 255435 ms
Total hot run time: 182999 ms

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 92.31% (24/26) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.68% (19540/37090)
Line Coverage 36.26% (182284/502668)
Region Coverage 32.57% (141286/433809)
Branch Coverage 33.62% (61270/182260)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 92.31% (24/26) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.42% (25957/36343)
Line Coverage 54.14% (271458/501419)
Region Coverage 51.40% (225226/438186)
Branch Coverage 52.95% (96887/182964)

@yiguolei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 29164 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b0dd8323507bf2f38cd4bb46113c6b79d304ba03, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17703	4466	4299	4299
q2	q3	10645	830	513	513
q4	4685	372	260	260
q5	7665	1211	1042	1042
q6	182	180	152	152
q7	835	833	680	680
q8	10312	1542	1441	1441
q9	5376	4782	4681	4681
q10	6872	1885	1643	1643
q11	507	289	263	263
q12	796	566	466	466
q13	17766	4231	3422	3422
q14	245	234	210	210
q15	938	791	786	786
q16	761	719	692	692
q17	721	941	435	435
q18	5962	5299	5319	5299
q19	1458	1004	645	645
q20	512	506	411	411
q21	4768	2034	1563	1563
q22	396	313	261	261
Total cold run time: 99105 ms
Total hot run time: 29164 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4771	4584	4615	4584
q2	q3	1810	2252	1784	1784
q4	904	1200	788	788
q5	4125	4418	4459	4418
q6	196	180	145	145
q7	1759	1641	1540	1540
q8	2494	2718	2646	2646
q9	7533	7496	7378	7378
q10	2620	2896	2439	2439
q11	494	439	401	401
q12	501	582	441	441
q13	4104	4517	3551	3551
q14	280	286	269	269
q15	875	790	814	790
q16	703	770	767	767
q17	1195	1536	1358	1358
q18	7117	6781	6518	6518
q19	1006	926	903	903
q20	2069	2150	2122	2122
q21	4011	3537	3445	3445
q22	463	452	409	409
Total cold run time: 49030 ms
Total hot run time: 46696 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183759 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b0dd8323507bf2f38cd4bb46113c6b79d304ba03, data reload: false

query5	4511	685	516	516
query6	338	243	230	230
query7	4220	474	271	271
query8	361	253	249	249
query9	8755	2783	2798	2783
query10	477	378	346	346
query11	17092	17486	17265	17265
query12	204	137	132	132
query13	1380	488	434	434
query14	6878	3366	3055	3055
query14_1	2966	2886	2912	2886
query15	225	196	185	185
query16	986	483	478	478
query17	1301	776	643	643
query18	3667	479	392	392
query19	254	221	183	183
query20	137	142	132	132
query21	227	164	126	126
query22	5077	4902	4883	4883
query23	17281	16762	16604	16604
query23_1	16666	16649	16600	16600
query24	7286	1648	1246	1246
query24_1	1230	1232	1238	1232
query25	569	475	437	437
query26	1241	265	160	160
query27	2741	495	294	294
query28	4480	1854	1879	1854
query29	797	562	466	466
query30	309	240	207	207
query31	909	704	626	626
query32	80	68	71	68
query33	500	339	281	281
query34	914	922	564	564
query35	639	663	594	594
query36	1140	1095	963	963
query37	131	94	78	78
query38	2974	2971	2862	2862
query39	892	860	847	847
query39_1	814	825	830	825
query40	231	154	135	135
query41	64	60	61	60
query42	103	107	99	99
query43	364	386	356	356
query44	
query45	196	184	184	184
query46	903	987	600	600
query47	2125	2128	2066	2066
query48	307	325	234	234
query49	629	469	372	372
query50	668	282	211	211
query51	4076	4065	4038	4038
query52	105	109	96	96
query53	296	331	279	279
query54	286	266	302	266
query55	88	93	79	79
query56	324	300	302	300
query57	1390	1329	1267	1267
query58	285	278	269	269
query59	2615	2693	2515	2515
query60	330	338	340	338
query61	151	139	146	139
query62	631	574	554	554
query63	315	274	280	274
query64	4899	1288	994	994
query65	
query66	1385	463	347	347
query67	16295	16216	16249	16216
query68	
query69	392	327	279	279
query70	985	942	921	921
query71	333	298	297	297
query72	2792	2691	2424	2424
query73	539	563	326	326
query74	10048	9907	9692	9692
query75	2873	2764	2446	2446
query76	2310	1024	690	690
query77	355	391	314	314
query78	11115	11436	10699	10699
query79	3152	824	608	608
query80	1784	614	544	544
query81	584	282	239	239
query82	1000	147	114	114
query83	327	258	247	247
query84	251	117	102	102
query85	877	472	448	448
query86	526	317	292	292
query87	3096	3131	2992	2992
query88	3606	2677	2677	2677
query89	435	376	346	346
query90	2179	168	167	167
query91	159	151	140	140
query92	79	77	73	73
query93	1827	858	501	501
query94	649	316	291	291
query95	592	403	318	318
query96	644	519	227	227
query97	2443	2467	2436	2436
query98	227	212	212	212
query99	1044	1034	922	922
Total cold run time: 258497 ms
Total hot run time: 183759 ms

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 86.67% (26/30) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.67% (19535/37090)
Line Coverage 36.23% (182123/502668)
Region Coverage 32.58% (141347/433807)
Branch Coverage 33.61% (61255/182258)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 93.33% (28/30) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.74% (26071/36343)
Line Coverage 54.42% (272868/501419)
Region Coverage 52.02% (227954/438184)
Branch Coverage 53.39% (97680/182962)

uchenily added a commit to uchenily/doris that referenced this pull request Feb 16, 2026
uchenily added a commit to uchenily/doris that referenced this pull request Feb 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants