feat(sampling): add locally typical sampling (typical_p)

TECHNICAL_REPORTS/1482-locally-typical-sampling-20260828.en.md

@@ -0,0 +1,46 @@
+# Technical Report: PR #1482 - Locally typical sampling (typical_p)
+
+**Date**: 2026-08-28
+
+**Status**: Completed
+
+**Languages**: Rust
+
+**Risk Level**: Medium
+
+## Executive Summary
+
+Adds locally typical sampling (`typical_p`, default `1.0` disabled) on the #1375 row-filter hook: tokens whose surprisal is closest to the row entropy are kept in typicality order until `typical_p` probability mass accumulates. Unlike the two earlier filters this one flips its two llama-compat manifest entries (`--typical`, `field:typical_p`) to `supported`, which pulled the server-wide flag on both binaries and the native `/completion` field into the same PR.
+
+## 1. Problem Statement
+
+mlxcel had no typical sampler, and the `--typical` / `field:typical_p` manifest entries are owned by issue #1377 itself: the epic's closure rule requires every entry owned by an issue to leave `deferred` before that issue closes, and the llama-compat gates tie a supported option claim to both server binaries and a supported field claim to `NativeCompletionRequest`. The scope therefore had to cover the engine semantics, both flag surfaces, the native field, and the manifest flip in one change.
+
+## 2. Technical Decisions
+
+### 2.1 Chain position: an idempotent top-k pre-mask
+
+The review caught the load-bearing subtlety: b10621's chain is `top_k -> typ_p` and its typical sampler re-softmaxes the surviving candidates, so entropy comes from the renormalized top-k distribution. mlxcel's hook runs before the C++ chain's top_k, so the filter initially computed entropy over the full vocabulary, diverging for essentially every enabled request (server default top_k 40). The fix masks the row to top-k ahead of the typical filter whenever `1 < top_k < vocab`; masking to top-k is idempotent, so the C++ chain's own top_k re-selects the same set and no stage runs out of order. A regression test uses a row where the two orders provably disagree (full-vocab typicality keeps only token 1, top-k-renormalized typicality keeps only token 0).
+
+### 2.2 Value domains split by surface
+
+OpenAI-shaped endpoints validate `(0.0, 1.0]` and return 400 otherwise. The native `/completion` field mirrors b10621's limit-free schema: a present out-of-domain value resolves to the explicit disabled `1.0`, still overriding the server default the way an upstream request value replaces it. The server-wide `--typical` flag accepts what b10621 accepts but folds out-of-domain values to disabled at startup with a warning, and `/props` reports the resolved value so the operator can read it back. The CLI generate/chat flag rejects out-of-domain values at parse time.
+
+### 2.3 Effective-value normalization and serde default
+
+`FusedSampleParams::from_config` and the speculative-window `sampling_config_eq` compare `effective_typical_p()` (greedy and out-of-domain fold to `1.0`), so necessarily-identical rows stay on the fused batch, pipelined lookahead, and shared speculative windows. The disaggregated wire struct defaults the field to `1.0` through a serde default fn, because the f32 zero default would turn an older peer's frame into an invalid always-on cutoff.
+
+## 3. Change Summary
+
+| Item | Value |
+|------|-------|
+| Files changed | 35 (+8 in the review follow-up) |
+| Lines | +884 / -33 across both commits |
+| Tests added | 25 |
+
+Validation: 91 core sampling tests green including a 40-case f64 host-reference test; compat integration test validates the flipped `--typical` claim against both freshly built binaries; manifest checker green with zero remaining #1377-owned deferred entries; real-checkpoint check on Qwen3-4B-4bit (fluent at `typical_p 0.5` with and without `top_k 40`, greedy byte-identical control).
+
+## 4. Follow-up Actions
+
+- #1373 (`p_less`) fills the remaining slot in `apply_row_filters`.
+- #1436 completes the remaining b10621 sampling semantics; the `+inf` handling difference between `typical_p_filter` (masked) and `top_n_sigma_filter` (kept) is documented in the filter docs.

TECHNICAL_REPORTS/1482-locally-typical-sampling-20260828.ko.md

@@ -0,0 +1,46 @@
+# 기술 보고서: PR #1482 - Locally typical sampling (typical_p)
+
+**작성일**: 2026-08-28
+
+**상태**: 완료
+
+**언어**: Rust
+
+**위험도**: Medium
+
+## 요약
+
+#1375 row-filter 훅 위에 locally typical sampling(`typical_p`, 기본값 `1.0` 비활성)을 추가했다. surprisal이 행 엔트로피에 가장 가까운 토큰부터 typicality 순서로 `typical_p` 확률 질량이 쌓일 때까지 남긴다. 앞의 두 필터와 달리 이 이슈가 소유한 llama-compat 매니페스트 엔트리 2건(`--typical`, `field:typical_p`)을 `supported`로 플립했고, 그 때문에 두 서버 바이너리의 서버 전역 플래그와 네이티브 `/completion` 필드가 같은 PR에 들어왔다.
+
+## 1. 문제 정의
+
+mlxcel에는 typical 샘플러가 없었고, `--typical` / `field:typical_p` 엔트리는 이슈 #1377 소유다. 에픽의 종료 규칙상 이슈가 닫히기 전에 소유 엔트리 전부가 `deferred`를 벗어나야 하고, llama-compat 게이트는 supported 옵션 주장을 두 서버 바이너리에, supported 필드 주장을 `NativeCompletionRequest`에 묶는다. 따라서 엔진 의미론, 두 플래그 표면, 네이티브 필드, 매니페스트 플립이 한 변경에 함께 들어가야 했다.
+
+## 2. 기술적 선택과 그 이유
+
+### 2.1 체인 위치: 멱등 top-k 프리마스크
+
+리뷰가 핵심 미묘함을 잡았다. b10621의 체인은 `top_k -> typ_p`이고 typical 샘플러는 생존 후보를 다시 softmax하므로 엔트로피가 재정규화된 top-k 분포에서 나온다. mlxcel 훅은 C++ 체인의 top_k보다 앞서 실행되어 처음에는 전체 어휘로 엔트로피를 계산했고, 서버 기본 top_k가 40이라 사실상 모든 활성 요청이 관찰 가능하게 갈라졌다. 수정은 `1 < top_k < vocab`일 때 typical 필터 앞에 top-k 마스크를 먼저 적용한다. top-k 마스킹은 멱등이라 C++ 체인의 top_k가 같은 집합을 다시 고르며 어떤 단계도 순서가 어긋나지 않는다. 두 순서가 증명 가능하게 갈라지는 행(전체 어휘 기준은 토큰 1만, top-k 재정규화 기준은 토큰 0만 남는다)으로 회귀 테스트를 추가했다.
+
+### 2.2 표면별 값 도메인 분리
+
+OpenAI 형태 엔드포인트는 `(0.0, 1.0]`을 검증하고 벗어나면 400을 반환한다. 네이티브 `/completion` 필드는 b10621의 무제한 스키마를 따라, 도메인 밖 값이 오면 명시적 비활성 `1.0`으로 해석하되 서버 기본값은 여전히 덮어쓴다. 서버 전역 `--typical` 플래그는 b10621처럼 받아들이되 시작 시 도메인 밖 값을 경고와 함께 비활성으로 접고, `/props`가 해석된 값을 보고해 운영자가 읽어볼 수 있다. CLI generate/chat 플래그는 파싱 시점에 거부한다.
+
+### 2.3 유효값 정규화와 serde 기본값
+
+`FusedSampleParams::from_config`와 speculative 윈도우의 `sampling_config_eq`는 `effective_typical_p()`(greedy와 도메인 밖 값은 `1.0`으로 접힘)를 비교하므로, 필연적으로 동일한 행들이 fused batch, pipelined lookahead, 공유 speculative 윈도우에 남는다. 분산 와이어 구조체는 serde 기본값 함수로 `1.0`을 쓴다. f32의 0 기본값이면 구버전 피어의 프레임이 항상 켜진 잘못된 컷오프가 되기 때문이다.
+
+## 3. 변경 요약
+
+| 항목 | 값 |
+|-----|---|
+| 변경 파일 | 35개 (리뷰 후속 +8) |
+| 라인 | 두 커밋 합산 +884 / -33 |
+| 테스트 추가 | 25개 |
+
+검증: 40케이스 f64 호스트 레퍼런스 테스트를 포함해 코어 sampling 테스트 91개 green, compat 통합 테스트가 새로 빌드한 두 바이너리에서 플립된 `--typical` 주장을 검증, 매니페스트 체커 green(#1377 소유 deferred 엔트리 0건), Qwen3-4B-4bit 실체크포인트 검증(`typical_p 0.5`에서 `top_k 40` 유무와 무관하게 유창, greedy 대조군 토큰 동일).
+
+## 4. 후속 조치
+
+- #1373(`p_less`)이 `apply_row_filters`의 남은 슬롯을 채운다.
+- #1436이 나머지 b10621 샘플링 의미론을 마무리한다. `typical_p_filter`(마스킹)와 `top_n_sigma_filter`(유지)의 `+inf` 처리 차이는 필터 문서에 기록해 두었다.

compat/llama-server/b10621/sampling-and-grammar.json

@@ -820,12 +820,20 @@
       "default": "1.00, 1.0 = disabled",
       "description": "locally typical sampling, parameter p (default: 1.00, 1.0 = disabled)",
       "discovery": "help-entry-head",
-      "state": "deferred",
+      "state": "supported",
       "issue": 1377,
-      "test": null,
-      "notes": null,
+      "test": "tests/llama_compat_manifest.rs",
+      "notes": "Server-wide default via --typical / --typical-p on both binaries (mlxcel serve and mlxcel-server), default 1.0 = disabled, matching b10621. Chain position matches b10621 (top_k -> typ_p -> top_p -> min_p): when top_k is active, an idempotent Rust-side top-k mask runs ahead of the typical filter so the typicality statistics come from the renormalized top-k survivors, and the fused chain's own top_k then re-selects the same set. An out-of-range startup value folds to disabled with a warning, mirroring the upstream sampler treating such values as inert.",
       "divergence": [],
-      "mlxcel": null
+      "mlxcel": {
+        "accepted_spellings": [
+          "--typical",
+          "--typical-p"
+        ],
+        "defaults": [
+          "1"
+        ]
+      }
     },
     {
       "kind": "option",
@@ -1462,12 +1470,14 @@
       "field_type": "num",
       "description": "Enable locally typical sampling with parameter p (1.0 = disabled)",
       "discovery": "source-schema-declaration",
-      "state": "deferred",
+      "state": "supported",
       "issue": 1377,
-      "test": null,
-      "notes": null,
+      "test": "src/server/llama_compat_tests.rs",
+      "notes": "b10621 declares the field with no schema limits; a present value outside the enabled range (0.0, 1.0) resolves to the explicit disabled form 1.0 and still overrides the server-wide default. Chain position matches b10621: typicality is computed over the renormalized top-k survivors (see the --typical entry).",
       "divergence": [],
-      "mlxcel": null
+      "mlxcel": {
+        "field": "typical_p"
+      }
     },
     {
       "kind": "native_request_field",

docs/python-client.md

@@ -109,7 +109,7 @@ print(llm.chat(messages, max_tokens=128, temperature=0.3))
 
 ## Sampling parameters
 
-Generation methods accept OpenAI sampling fields directly: `max_tokens`, `temperature`, `top_p`, `stop`, `seed`, `presence_penalty`, `frequency_penalty`, `logit_bias`, and `response_format`. Server-specific knobs that are not part of the OpenAI schema (`top_k`, `min_p`, `top_n_sigma`, `repetition_penalty`, DRY settings, `xtc_probability` / `xtc_threshold`, and the vLLM-compatible loop-detection fields `max_pattern_size` / `min_pattern_size` / `min_count`) are forwarded in the request body. `xtc_probability` (`0.0`-`1.0`, default `0.0`, disabled) is the per-step chance that XTC (Exclude Top Choices) removes all but the single least-probable token among those whose probability exceeds `xtc_threshold` (`0.0`-`0.5`, default `0.1`). The threshold is tested against the renormalised distribution that remains after `top_k`, `top_p` and `min_p` have run, not against the raw model distribution, matching the llama-server chain; both are validated on `/v1/chat/completions`, `/v1/completions`, and `/v1/responses`, and an out-of-range value returns a 400 before generation starts. `top_n_sigma` (default `0.0`, disabled) keeps only the tokens whose raw logit lies within N standard deviations of the row maximum before the temperature / top-k / top-p / min-p chain runs, which keeps sampling stable at high temperature; it must be finite and `>= 0.0` on the OpenAI-shaped endpoints (a negative value returns a 400). The native `/completion` spelling of the field lands together with its llama-server compatibility-manifest entry (#1436). You can also pass an explicit `extra_body={...}` for arbitrary server fields; values you set there win on conflict.
+Generation methods accept OpenAI sampling fields directly: `max_tokens`, `temperature`, `top_p`, `stop`, `seed`, `presence_penalty`, `frequency_penalty`, `logit_bias`, and `response_format`. Server-specific knobs that are not part of the OpenAI schema (`top_k`, `min_p`, `top_n_sigma`, `typical_p`, `repetition_penalty`, DRY settings, `xtc_probability` / `xtc_threshold`, and the vLLM-compatible loop-detection fields `max_pattern_size` / `min_pattern_size` / `min_count`) are forwarded in the request body. `xtc_probability` (`0.0`-`1.0`, default `0.0`, disabled) is the per-step chance that XTC (Exclude Top Choices) removes all but the single least-probable token among those whose probability exceeds `xtc_threshold` (`0.0`-`0.5`, default `0.1`). The threshold is tested against the renormalised distribution that remains after `top_k`, `top_p` and `min_p` have run, not against the raw model distribution, matching the llama-server chain; both are validated on `/v1/chat/completions`, `/v1/completions`, and `/v1/responses`, and an out-of-range value returns a 400 before generation starts. `top_n_sigma` (default `0.0`, disabled) keeps only the tokens whose raw logit lies within N standard deviations of the row maximum before the temperature / top-k / top-p / min-p chain runs, which keeps sampling stable at high temperature; it must be finite and `>= 0.0` on the OpenAI-shaped endpoints (a negative value returns a 400). The native `/completion` spelling of the field lands together with its llama-server compatibility-manifest entry (#1436). `typical_p` (default `1.0`, disabled) enables locally typical sampling: tokens whose surprisal is closest to the row entropy are kept, in typicality order, until `typical_p` probability mass accumulates; it must be finite and in `(0.0, 1.0]` on the OpenAI-shaped endpoints (400 otherwise), the native `/completion` field mirrors llama-server's limit-free schema by resolving a present out-of-range value to the explicit disabled form, and the server-wide `--typical` / `--typical-p` flag sets the default applied when a request omits the field. You can also pass an explicit `extra_body={...}` for arbitrary server fields; values you set there win on conflict.
 
 ```python
 llm.generate("Once upon a time", max_tokens=200, top_p=0.9, top_k=40, min_p=0.05)

src/bin/bench_decode.rs

@@ -297,6 +297,7 @@ fn sampling_config(model_path: &Path) -> SamplingConfig {
         xtc_probability: 0.0,
         xtc_threshold: 0.1,
         top_n_sigma: 0.0,
+        typical_p: 1.0,
         stop_token_ids: mlxcel::read_eos_token_ids(model_path),
     })
 }

src/bin/mlx_server.rs

@@ -966,6 +966,15 @@ struct ServerArgs {
     #[arg(long = "min-p", default_value_t = 0.05)]
     min_p: f32,
 
+    /// Default locally typical sampling, parameter p (1.0 = disabled)
+    #[arg(
+        long = "typical",
+        alias = "typical-p",
+        value_name = "N",
+        default_value_t = 1.0
+    )]
+    typical_p: f32,
+
     /// Random seed (-1 = random)
     #[arg(short = 's', long = "seed", default_value_t = -1)]
     seed: i64,
@@ -2095,6 +2104,7 @@ fn build_startup_input(mut args: ServerArgs) -> anyhow::Result<ServerStartupInpu
         top_p: args.top_p,
         top_p_was_set: long_cli_flag_was_set("top-p"),
         min_p: args.min_p,
+        typical_p: args.typical_p,
         seed: args.seed,
         repeat_last_n: args.repeat_last_n,
         repeat_penalty: args.repeat_penalty,

src/commands/chat_tests.rs

@@ -206,6 +206,7 @@ fn chat_options_new_sets_conventional_defaults() {
         xtc_probability: 0.0,
         xtc_threshold: 0.1,
         top_n_sigma: 0.0,
+        typical_p: 1.0,
         stop_token_ids: Vec::new(),
     };
     let opts = ChatOptions::new(PathBuf::from("models/foo"), 128, sampling);

src/commands/generate.rs

@@ -956,6 +956,7 @@ fn resolved_cli_sampling_params(
         xtc_probability: 0.0,
         xtc_threshold: 0.1,
         top_n_sigma: args.sampling.top_n_sigma,
+        typical_p: args.sampling.typical_p,
         stop_token_ids,
     }
 }

src/commands/generate_tests.rs

@@ -563,6 +563,7 @@ fn sample_generate_args(model_path: PathBuf) -> crate::GenerateArgs {
             top_k: 0,
             min_p: 0.0,
             top_n_sigma: 0.0,
+            typical_p: 1.0,
             repetition_penalty: 1.0,
             dry_multiplier: 0.0,
             dry_base: 1.75,

src/commands/serve.rs

@@ -514,6 +514,7 @@ fn build_startup_input(mut args: crate::ServeArgs) -> anyhow::Result<ServerStart
         top_p: args.top_p,
         top_p_was_set: long_cli_flag_was_set("top-p"),
         min_p: args.min_p,
+        typical_p: args.typical_p,
         seed: args.seed,
         repeat_last_n: args.repeat_last_n,
         repeat_penalty: args.repeat_penalty,

src/commands/serve_tests.rs

@@ -87,6 +87,7 @@ fn sample_args() -> crate::ServeArgs {
         top_k: 50,
         top_p: 0.95,
         min_p: 0.2,
+        typical_p: 1.0,
         seed: -1,
         repeat_last_n: 32,
         repeat_penalty: 1.2,

src/distributed/disaggregated/decode_scheduler_tests.rs

@@ -21,6 +21,7 @@ fn default_sampling() -> SerializableSamplingState {
         frequency_penalty: 0.0,
         presence_penalty: 0.0,
         top_n_sigma: 0.0,
+        typical_p: 1.0,
         stop_token_ids: Vec::new(),
     }
 }

src/distributed/disaggregated/prefill_scheduler_tests.rs

@@ -23,6 +23,7 @@ fn default_sampling() -> SerializableSamplingState {
         frequency_penalty: 0.0,
         presence_penalty: 0.0,
         top_n_sigma: 0.0,
+        typical_p: 1.0,
         stop_token_ids: Vec::new(),
     }
 }

src/distributed/disaggregated/serving_protocol.rs

@@ -246,6 +246,7 @@ pub fn sampling_to_serializable(config: &SamplingConfig) -> SerializableSampling
         frequency_penalty: config.frequency_penalty,
         presence_penalty: config.presence_penalty,
         top_n_sigma: config.top_n_sigma,
+        typical_p: config.typical_p,
         stop_token_ids: config.stop_token_ids.clone(),
     }
 }
@@ -270,6 +271,7 @@ pub fn sampling_from_serializable(state: &SerializableSamplingState) -> Sampling
         frequency_penalty: state.frequency_penalty,
         presence_penalty: state.presence_penalty,
         top_n_sigma: state.top_n_sigma,
+        typical_p: state.typical_p,
         stop_token_ids: state.stop_token_ids.clone(),
         token_bias: Default::default(),
         // Loop detection is not yet serialized across the disaggregated

src/distributed/disaggregated/serving_protocol_tests.rs

@@ -219,6 +219,7 @@ fn sampling_round_trips_through_the_serializable_mirror() {
     config.stop_token_ids = vec![13, 14];
     config.dry_sequence_breakers = vec![198];
     config.top_n_sigma = 1.5;
+    config.typical_p = 0.5;
 
     let restored = sampling_from_serializable(&sampling_to_serializable(&config));
 
@@ -233,6 +234,7 @@ fn sampling_round_trips_through_the_serializable_mirror() {
     assert_eq!(restored.stop_token_ids, vec![13, 14]);
     assert_eq!(restored.dry_sequence_breakers, vec![198]);
     assert_eq!(restored.top_n_sigma, 1.5);
+    assert_eq!(restored.typical_p, 0.5);
 }
 
 #[test]
@@ -251,3 +253,19 @@ fn sampling_state_from_an_older_peer_without_top_n_sigma_defaults_to_disabled()
     assert_eq!(state.top_n_sigma, 0.0);
     assert_eq!(sampling_from_serializable(&state).top_n_sigma, 0.0);
 }
+
+#[test]
+fn sampling_state_from_an_older_peer_without_typical_p_defaults_to_disabled() {
+    // The serde default must be the DISABLED value 1.0, not the f32 zero
+    // default, which would be an invalid always-on cutoff.
+    let mut value = serde_json::to_value(sampling_to_serializable(&SamplingConfig::greedy()))
+        .expect("serialize sampling state");
+    value
+        .as_object_mut()
+        .expect("sampling state serializes as an object")
+        .remove("typical_p");
+    let state: SerializableSamplingState =
+        serde_json::from_value(value).expect("deserialize legacy sampling state");
+    assert_eq!(state.typical_p, 1.0);
+    assert_eq!(sampling_from_serializable(&state).typical_p, 1.0);
+}

src/distributed/kv_cache_serde/tests.rs

@@ -202,6 +202,7 @@ fn full_state_serialize_deserialize_round_trip() {
             frequency_penalty: 0.0,
             presence_penalty: 0.0,
             top_n_sigma: 0.0,
+            typical_p: 1.0,
             stop_token_ids: vec![2],
         }),
         token_history: vec![100, 200, 300],