Skip to content

Feature Request: Make llm_arch_supports_mixed_batch suitable for C and FFI consumers #30026

Open

Description

@ardan-bkennedy

Prerequisites

  • I am running the latest code. Mention the version if possible as well.
  • I carefully followed the README.md.
  • I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
  • I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

I need access to this API. Here are some details:

  • PR llama: support both embd + raw tokens in batch #29622 added support for batches containing both raw token entries and embedding entries.
  • Some model architectures do not support these mixed batches.
  • llama.cpp checks this internally with llm_arch_supports_mixed_batch.
  • llm_arch_supports_mixed_batch is not part of the public C API and takes the private llm_arch type, so Go/libffi consumers cannot safely bind it.
  • Request a supported C API that reports whether a loaded model supports mixed token-and-embedding batches.

Here is some related work and sources

Motivation

Here are my motivations

  • Yzma binds llama_batch_ext and llama_process through Go/libffi.
  • Yzma needs to know whether a loaded model supports a mixed batch before submitting one.
  • The internal C++ helper cannot be safely bound because it is not public C API and uses the private llm_arch type.
  • Copying llama.cpp's architecture exception list into Yzma would become stale as architectures are added or changed.
  • A public model capability query would let Yzma expose the same decision safely to Go callers

Possible Implementation

Here are some ideas for a possible implementation

A small public C wrapper around the existing internal check, following the existing model capability APIs:

LLAMA_API bool llama_model_supports_mixed_batch(
        const struct llama_model * model);

The implementation can call llm_arch_supports_mixed_batch(model->arch) internally. The exact API name belongs to llama.cpp maintainers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions