[ggerganov]

alt #19747
fix #19690
fix #19858

This continues the work from #19747 to enable multi-modal checkpointing in llama-server. Avoids changes to libllama. Fixes n_past ambiguity of "number of tokens" and "next position" by tracking the position separately with a new pos_next var.

[timkhronos]

I think it might make more sense to post this here instead of the original pr:

@ggerganov These 3 changes should fix the incorrect checkpoint reuse I mentioned in regards to #19849

in server-task.h, add a new pos_end to checkpoint struct:

struct server_prompt_checkpoint {
    llama_pos pos_min;
    llama_pos pos_max;
    llama_pos pos_end; //full prompt extent when checkpoint was created

    std::vector<uint8_t> data;

in sever-context.cpp at checkpoint creation:

auto & cur = slot.prompt.checkpoints.emplace_back(server_prompt_checkpoint{
    /*.pos_min = */ pos_min,
    /*.pos_max = */ pos_max,
    /*.pos_end = */ slot.prompt.tokens.pos_next(),
    /*.data   = */ std::vector<uint8_t>(checkpoint_size),
});

and also in server-context.cpp, when checking if we have valid checkpoints:

const llama_pos prompt_end = slot.task->tokens.pos_next();

const auto it = std::find_if(
    slot.prompt.checkpoints.rbegin(),
    slot.prompt.checkpoints.rend(),
    [&](const auto & cur) {
        if (cur.pos_end > prompt_end) {
            return false;
        }
        // guarantee that a checkpoint will result in at least one token being processed [TAG_PROMPT_LOGITS]
        return cur.pos_min < pos_min_thold;
    }
);

[ggerganov]

@ngxson PTAL, I think this is good to merge.

[ngxson]

should we change this to int64_t to match what's pos_next(int64_t) expects?