Significantly higher gpu memory consumption in koboldcpp 1.98.1 compared to 1.95.1 when using both embedding model and text model

Describe the Issue
After updating to 1.98.1 from 1.95.1, attempting to run the same model configuration (text model + embedding) results in a crash because of not having enough gpu memory.

Having attempted to reduce the number of offloaded GPU layers I have noticed that even with 0 gpu layers offloaded it still uses 5 gb of gpu memory, against 700 mb with version 1.95.1

The 4gb+ difference disappears if I remove either the embedding model or the text model.

The embedding model is available at

https://huggingface.co/pqnet/bge-m3-Q8_0-GGUF

Additional Information:
Platform is Linux/CUDA 12.8
configuration file:

{
  "model": [
    "Llama-3-8B-Q4_K_S.gguf"
  ],
  "model_param": "Llama-3-8B-Q4_K_S.gguf",
  "port": 5001,
  "port_param": 5001,
  "host": "192.168.112.61",
  "launch": false,
  "config": null,
  "threads": 7,
  "usecublas": [
    "mmq"
  ],
  "usevulkan": null,
  "useclblast": null,
  "usecpu": false,
  "contextsize": 8192,
  "gpulayers": 32,
  "tensor_split": null,
  "version": false,
  "analyze": "",
  "ropeconfig": [
    0.0,
    10000.0
  ],
  "blasbatchsize": 512,
  "blasthreads": 0,
  "lora": null,
  "noshift": false,
  "nofastforward": false,
  "usemmap": true,
  "usemlock": false,
  "noavx2": false,
  "failsafe": false,
  "debugmode": 0,
  "onready": "",
  "benchmark": null,
  "prompt": "",
  "promptlimit": 100,
  "multiuser": 1,
  "multiplayer": false,
  "websearch": true,
  "remotetunnel": false,
  "highpriority": false,
  "foreground": false,
  "preloadstory": "",
  "savedatafile": "",
  "quiet": false,
  "ssl": null,
  "nocertify": false,
  "mmproj": "",
  "visionmaxres": 1024,
  "draftmodel": "",
  "draftamount": 8,
  "draftgpulayers": 999,
  "draftgpusplit": null,
  "password": null,
  "ignoremissing": false,
  "chatcompletionsadapter": "AutoGuess",
  "flashattention": false,
  "quantkv": 1,
  "forceversion": 0,
  "smartcontext": false,
  "unpack": "",
  "exporttemplate": "",
  "embeddingsmodel": "bge-m3-Q8_0.gguf",
  "embeddingsgpu": false,
  "nomodel": false,
  "moeexperts": -1,
  "showgui": false,
  "skiplauncher": true,
  "hordemodelname": "",
  "hordeworkername": "",
  "hordekey": "",
  "hordemaxctx": 0,
  "hordegenlen": 0,
  "sdmodel": "",
  "sdthreads": 0,
  "sdclamped": 0,
  "sdt5xxl": "",
  "sdclipl": "",
  "sdclipg": "",
  "sdvae": "",
  "sdvaeauto": false,
  "sdquant": false,
  "sdlora": "",
  "sdloramult": 1.0,
  "sdnotile": false,
  "whispermodel": "",
  "ttsmodel": "",
  "ttswavtokenizer": "",
  "ttsgpu": false,
  "ttsmaxlen": 4096,
  "ttsthreads": 0,
  "admin": true,
  "adminpassword": null,
  "admindir": "",
  "hordeconfig": null,
  "sdconfig": null,
  "noblas": false,
  "nommap": false
}

Hmm... the context length of loaded embedding models now defaults to the same context length of loaded text models. You can override this with --embeddingsmaxctx, does it help?

I have restricted a bit more the reproduction surface. Apparently this only happens if flashAttention is disabled.

Hmm... the context length of loaded embedding models now defaults to the same context length of loaded text models. You can override this with --embeddingsmaxctx, does it help?

Adding embeddingsmaxctx: 8192 to the configuration file made no difference. still ~4.5 gb more gpu memory allocated for 1.98.1 compared to 1.95.1 (with gpulayers: 1 I get 918 mb vs 5478 mb from nvidia-smi, and with all layers it still crashes)

Note that the embedding model has a default context of 8192 tokens, same as the configured contextsize for the text model.

If i use embeddingsmaxctx: 1024 instead the difference (between the two versions) becomes much smaller: 1022 mb vs 918 mb.

Should it make any difference though if we use embeddingsgpu: false?

I have restricted a bit more the reproduction surface. Apparently this only happens if flashAttention is disabled.

I must correct myself here. With flashAttention enabled there is still some difference in gpu memory usage, although it is significantly lower (1428 mb vs 898 mb with full 8k context and gpulayers: 1, difference of about 500 mb), this may suggest the extra memory usage is related to the logits or kv cache (which is quantized differently with flash attention)

Alright i'll look into it

Hi, can you try 1.99, should be fixed.

Hi, can you try 1.99, should be fixed.

Thank you for looking into this. Pacman currently botched my grub installation, I'll test it as soon as I can find an usb stick to boot my linux machine again -.-

@LostRuins

@LostRuins tested with 1.99.4 GPU memory usage is down to 900mb with flash attention and 934mb without it, the same as 1.95.1 (it too has a slightly higher memory usage footprint now, possibly different than before because I have updated CUDA and gpu drivers in the meanwhile). Issue seems solved for me.

for tracking, original issue ggml-org#15586

attempted fix: 7b4517c#diff-9c76f92045438deae73cc18c608142fddd561d05228fcff1e51e4212061b0e86

Navigation Menu

Repository navigation

Significantly higher gpu memory consumption in koboldcpp 1.98.1 compared to 1.95.1 when using both embedding model and text model #1734

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Significantly higher gpu memory consumption in koboldcpp 1.98.1 compared to 1.95.1 when using both embedding model and text model #1734

Description

Activity

LostRuins commented on Sep 11, 2025

pqnet commented on Sep 14, 2025

pqnet commented on Sep 14, 2025

pqnet commented on Sep 14, 2025

LostRuins commented on Sep 14, 2025

LostRuins commented on Sep 21, 2025

pqnet commented on Sep 23, 2025

pqnet commented on Sep 28, 2025

LostRuins commented on Jan 18, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions