[ngxson]

Overview

Support 5 decision models: laya, julia-1, lev, openjev (+vision), kev

Pre-converted GGUF:

llama-server -hf ggml-org/OpenJev-GGUF
llama-server -hf ggml-org/lev-GGUF
llama-server -hf ggml-org/Laya-GGUF
llama-server -hf ggml-org/Julia-1-GGUF
llama-server -hf ggml-org/Kev-4B-GGUF

Additional information

"Decision models" are just fancy wrappers around traditional embedding models (BERT/Qwen/etc), so changes to support them are kept to be as small and as self-contained as possible.

Since the underlay infra is just existing embedding models, changes to libllama are minimal. Main part is server_decision_context that switches the input/output handling based on {arch}.decision.type gguf metadata.

Recap changes per-component:

• Conversion: handle conversion + new tensors/metadata + chat template injection
• libllama: add decision head on graph, add some getters API
• Server: add /v1/systemone API, add the main server_decision_context handling logic, add tests (using a slice of laya model)

TODO:

• maybe support https://huggingface.co/Cloudflare/clef , just out 2 hours ago at the time writing this planned as a follow-up
• add docs (+ dev docs for those who will add future model support)
• support parallel shared prompt prefix (only for causal models)
• support vision input

Testing

Table represent result in format: reference → llama.cpp

input json

{
  "state": {
    "message": "Hi, I was charged twice for my order #4471 and I want a refund.",
    "plan": "pro",
    "order": {
      "id": 4471,
      "items": ["phone case", "charger"]
    }
  },
  "questions": {
    "intent": {
      "type": "choice",
      "instructions": "What does the customer want?",
      "criteria": {
        "refund": "wants money back",
        "cancel": "wants to cancel an order",
        "track": "wants to know where an order is",
        "other": "anything else"
      }
    },
    "urgent": {
      "type": "noul",
      "instructions": "Does this need a human within the hour?"
    },
    "frustration": {
      "type": "score",
      "instructions": "How frustrated is the customer?",
      "criteria": ["calm", "mildly annoyed", "annoyed", "angry"]
    },
    "refund": {
      "type": "noul",
      "instructions": "Is a refund requested?",
      "criteria": {
        "true": "money back is asked",
        "false": "no money back is asked"
      }
    },
    "team": {
      "type": "choice",
      "instructions": "Which team?",
      "criteria": {
        "billing": null,
        "shipping": null,
        "technical": null,
        "sales": null,
        "legal": null,
        "returns": null,
        "fraud": null,
        "accounts": null,
        "retention": null,
        "other": null
      }
    }
  }
}

question	laya	julia-1	lev	openjev	kev-4b
intent (top option, refund)	0.9871 → 0.9870	0.9899 → 0.9910	0.8159 → 0.8209	0.9994 → 0.9994	0.6537 → 0.6550
urgent (noul)	0.1013 → 0.1016	0.6447 → 0.6232	0.4680 → 0.4681	0.2505 → 0.2507	0.7051 → 0.7045
frustration (top level)	0.3492 → 0.3492	0.6749 → 0.6895	0.2627 → 0.2633	0.5295 → 0.5295	0.4406 → 0.4407
refund (noul)	0.8916 → 0.8916	0.5608 → 0.5636	0.9503 → 0.9505	0.9901 → 0.9901	0.9472 → 0.9474
team (top option)	0.9666 → 0.9664	not run	0.3815 → 0.3787	0.9874 → 0.9874	0.4598 → 0.4599
worst diff, any probability	8.4e-4	2.2e-2	5.2e-3	1.8e-4	1.3e-3
input tokens	414 = 414	not reported by ref	576 vs 1194	619 = 619	406 = 406

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: most of the code is AI-written, I own the design

[ggerganov]

Is this correct "lev" instead of "kev" here?

[ngxson]

it's expected, they are quite similar so I grouped them into one file