Skip to content

info: cached info message - #21

Merged
ggerganov merged 4 commits into
ggml-org:masterfrom
VJHack:cached-info-message
Jan 6, 2025
Merged

ggerganov merged 4 commits into
ggml-org:masterfrom
VJHack:cached-info-message

Conversation

@VJHack

@VJHack VJHack commented Jan 5, 2025 •

Copy link
Copy Markdown
Contributor

Instead of displaying the original response metrics when the completion was pulled from cache, we want to display metrics related to the cache. The completion info is shown in the following format:

C: number of elements currently in the cache / cache size
t: total FIM time

 ... world\n");     | C: 3 / 250 | t: 0.66 ms

This is what the info message looks like when there is a cache hit.
Screen Shot 2025-01-05 at 2 42 25 AM

Changes in this PR:

  • Changed info message for when there is a cache hit
  • Optimized cache usage by only storing the content and not the other metrics. This helps reduce the size of the cache.

This suggestion was mentioned here: #18 (review)

@ggerganov ggerganov left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The llama-server now has the option to filter response fields, so another idea for improvement it to use this feature to minimize the amount of information that is transported from the server to the client, by filtering just the fields that are needed by llama.vim:

ggml-org/llama.cpp#10819

@ggerganov
ggerganov merged commit 3cc84b0 into ggml-org:master Jan 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants