Skip to content

mtmd: support dots3-note vision+audio - #27524

Merged
ngxson merged 6 commits into
masterfrom
xsn/dots3-note-mtmd
Aug 22, 2026
Merged

ngxson merged 6 commits into
masterfrom
xsn/dots3-note-mtmd

Conversation

@ngxson

@ngxson ngxson commented Aug 22, 2026 •

Copy link
Copy Markdown
Collaborator

Overview

Follow-up #27060

Add MoE support for vision tower

Audio support is trivial, just good old whisper

Compare logits vs HF version:

test tokens NMSE worst token cos
image 448x336 192 = HF 1.6e-4 0.9982
audio 7.3s 92 = HF 1.6e-6 0.999999
audio 65s (chunked) 750+63 = HF 1.5e-6 0.999999

Requirements

ngxson added 5 commits August 14, 2026 12:13
# Conflicts:
#	conversion/dots3.py
#	src/llama-graph.h
#	src/llama-model-saver.cpp
#	src/models/dots3note.cpp
#	tests/test-llama-archs.cpp
@ngxson
ngxson requested review from a team and CISC as code owners August 22, 2026 00:17
@github-actions github-actions Bot added mtmd Related to multimodal functionality (video/image/audio) conversion labels Aug 22, 2026
Comment thread gguf-py/gguf/tensor_mapping.py Outdated
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
@ngxson
ngxson merged commit 54ee5ee into master Aug 22, 2026
26 of 29 checks passed
therealkenc pushed a commit to therealkenc/llama.cpp that referenced this pull request Aug 24, 2026
* text: conversion

* init impl

* mtmd: conversion

* impl mtmd cpp

* Update gguf-py/gguf/tensor_mapping.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
thecodacus pushed a commit to thecodacus/llama.cpp that referenced this pull request Sep 7, 2026
* text: conversion

* init impl

* mtmd: conversion

* impl mtmd cpp

* Update gguf-py/gguf/tensor_mapping.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
zbrad pushed a commit to zbrad/llama.cpp that referenced this pull request Sep 10, 2026
* text: conversion

* init impl

* mtmd: conversion

* impl mtmd cpp

* Update gguf-py/gguf/tensor_mapping.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

conversion mtmd Related to multimodal functionality (video/image/audio)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants