Implement stochastic speculative sampling - #5625

Merged

ggerganov merged 17 commits into

ggml-org:masterfrom

mscheong01:stochastic-speculative-decoding

Mar 4, 2024

Commits on Feb 21, 2024

(WIP) Implement stochastic speculative decoding
mscheong01
committed
c1bad4a

Commits on Feb 22, 2024

sample from residual distribution on draft accept failure
mscheong01
committed
a9335a5
fix #5657: force greedy sampling with probs when temp is 0
mscheong01
committed
4694edd

Commits on Feb 27, 2024

remove p_accept parameter
mscheong01
committed
fb18827
fix style
mscheong01
committed
34b942a
remove unused variables
mscheong01
committed
875319b
add srand() in speculative.cpp
mscheong01
committed
6afc1f6

Commits on Feb 28, 2024

replace use of rand() with mt19937 sampling
mscheong01
committed
94f6256
fixes based on review (@JohannesGaessler)
mscheong01
committed
e4896e7

Commits on Feb 29, 2024

Commits on Mar 2, 2024

Commits on Mar 4, 2024

remove malloc code by utilizing vectors
mscheong01
committed
67ad517
add PR link to README
mscheong01
committed
056bdb3