Misc. bug: Can't use longer context than model via RoPE due to server-imposed restrictions

[woof-dog]

Name and Version

version: 716 (10e9780)
built with cc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 for x86_64-linux-gnu

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

llama-server -m ./model/with/4096context -c 16384 --rope-scaling yarn --rope-scale 4

Problem description & steps to reproduce

The llama-server does not allow using the extended context. Server output announces the context length is capped

the slot context (%d) exceeds the training context of the model (%d) - capping\n

This is not aware of RoPE settings or other user configurations which would allow for longer context. Disabling this check in llama-server introduced in cd5e3b5 allowed me to use longer context via the RoPE settings.

This "capping" forces the model to load with 4,096 tokens of context and causes my long-context queries to fail.

Please allow us to override this cap. For users who don't know what they are doing, sure, it's probably helpful, but for advanced users there should be a way to disable it.

First Bad Commit

cd5e3b5

Relevant log output

the slot context (%d) exceeds the training context of the model (%d) - capping\n

[CISC]

You can, with --override-kv <insert architecture here>.context_length=int:16384.

[woof-dog]

Thank you for offering a solution. I just wish it were not so cumbersome (now I have to fish out the architecture code name and memorize this format). I instinctively looked through the help page when I first noticed the issue. It would be nice if we could add a flag to disable the cap. I'd be happy to write up the change into a PR if desired.

[CISC]

Well, the problem is really that the model is missing the correct YaRN metadata, I would suggest permanently adding those to the GGUF instead, using GGUF Editor or gguf-editor-gui.

[woof-dog]

Oh, it's not the GGUF that is wrong (it was indeed only trained with a context length of 4,096). I was testing out different YaRN/RoPE scaling parameters.

[felladrin]

I've been through the same problem when trying --rope-scaling yarn --rope-scale 4 recently. The arguments are ignored.

Well, the problem is really that the model is missing the correct YaRN metadata, I would suggest permanently adding those to the GGUF instead, using GGUF Editor or gguf-editor-gui.

But if the metadata has to exist in the GGUF, it makes the arguments --rope-scaling and --rope-scale useless, no? I mean, the arguments should allow us to force the model to behave in a specific way, exactly for the cases when the creator of the GGUF forgot to add to the metadata.

You can, with --override-kv .context_length=int:16384.

This worked for me too, but I guess the model wouldn't work properly beyond the original context if it's not correctly detecting the intention of extending the context. I wasn't able to confirm it.

To conclude, as the OP, I also think the --rope-scaling and --rope-scale need a review.

[CISC]

I've been through the same problem when trying --rope-scaling yarn --rope-scale 4 recently. The arguments are ignored.

They are not ignored, they do what they are supposed to do (they do not merely extend context), however there is currently a hard check that you are not exceeding the context length set in metadata.

You can, with --override-kv .context_length=int:16384.

This worked for me too, but I guess the model wouldn't work properly beyond the original context if it's not correctly detecting the intention of extending the context. I wasn't able to confirm it.

The rope options are not really meant for normal users, they significantly alter the behavior of the model, this is why it's best that the correct information is embedded in the GGUF in the first place.

[felladrin]

They are not ignored, they do what they are supposed to do (they do not merely extend context), however there is currently a hard check that you are not exceeding the context length set in metadata.

Maybe then a solution for this issue could be improving the feedback for the case when it exceeds the model's context_length. Just to let the user have a confirmation that the arguments were validated, but, due to the wrong metadata on the GGUF, it won't have the expected effect.

[CISC]

They are not ignored, they do what they are supposed to do (they do not merely extend context), however there is currently a hard check that you are not exceeding the context length set in metadata.

Maybe then a solution for this issue could be improving the feedback for the case when it exceeds the model's context_length. Just to let the user have a confirmation that the arguments were validated, but, due to the wrong metadata on the GGUF, it won't have the expected effect.

Probably a good idea.

[woof-dog]

The rope options are not really meant for normal users, they significantly alter the behavior of the model, this is why it's best that the correct information is embedded in the GGUF in the first place.

In my case, the GGUF had correct metadata. It was trained on exactly the rope values that are present in the GGUF. However, I wanted to see if I could squeeze additional context out of the model by increasing the rope value beyond what the model defaults are. And, with yarn, it worked.

Online, people have suggested to me in several places to manipulate the rope value for various effects, for example to have the model output more text than it usually would. My own usage, playing around with different rope values to see if I could get a reasonable higher context length without too much coherence degradation, I think is a good example showing that this is not beyond the scope of the usage from a "normal" user.

The model I was testing on showed coherence up to 4x beyond the value it was trained on. Again, the GGUF metadata was correct, the model wasn't supposed to go beyond 4,096 tokens. All that would need to be done is to add an option to override this capping of the context length which was added last month: cd5e3b5#diff-a18a8e64e12a01167d8e98fc545d9264ae1cfa8b846c1cccf0d4eed09d76d879R2504

[woof-dog]

@CISC I am frequently having to edit the server source files manually wherever I use llama.cpp when trying to experiment with different rope configurations. If I wrote a PR that adds a server CLI flag which, when used, skips the context length cap, would it be approved? Note that this cap only exists in llama-server.

[CISC]

Without having looked too deeply into it, I don't think it needs another CLI flag, the relevant server code should probably just check that the training context matches up with yarn settings, alternatively (and probably better) the training context should just be adjusted when there is no relevant metadata and you manually change the yarn settings.

[woof-dog]

Correct me if I read this wrong, but do you mean in your alternative that the server code should not enforce a cap on the context length when you manually change the rope or yarn settings? I would definitely be in favor of that if so, and could take a crack at the implementation.

[CISC]

Correct me if I read this wrong, but do you mean in your alternative that the server code should not enforce a cap on the context length when you manually change the rope or yarn settings? I would definitely be in favor of that if so, and could take a crack at the implementation.

Sort of, the cap should perhaps be left as-is, but if one changes yarn settings there should be a best-effort of readjusting the trained context size depending on whether unscaled context size was defined in metadata or not.