Skip to content

Implement lempel-ziv-trajectory-complexity grader (Tier 1, rank 11) #56972

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pelikhan merged 2 commits into from
Aug 29, 2026
Merged

Implement lempel-ziv-trajectory-complexity grader (Tier 1, rank 11) #56972

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c8072e6
Initial plan
Copilot Aug 29, 2026
2fa985d
Implement lempel-ziv-trajectory-complexity grader
Copilot Aug 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/shared/graders/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ to `Implemented` in the same PR that adds `shared/graders/<id>.md`.
| 8 | `recurrence-trapping-time` | Canonical states | Implemented |
| 9 | `recurrence-rate` | Canonical states | Implemented |
| 10 | `event-entropy-rate` | Event sequence only | Implemented |
| 11 | `lempel-ziv-trajectory-complexity` | Event sequence only | Not started |
| 11 | `lempel-ziv-trajectory-complexity` | Event sequence only | Implemented |

## Tier 2 — needs explicit constraints/states/provenance/objectives

Expand Down
120 changes: 120 additions & 0 deletions .github/workflows/shared/graders/lempel-ziv-trajectory-complexity.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
---
graders:
# LZ76 (Kaspar & Schuster) incremental-parsing complexity of the ordered
# canonical event-symbol sequence, projected purely from the Trajectory
# IR's events[] (no states, provenance, or objectives required).
# tool_call events are qualified by their ref (finer alphabet
# granularity, matching the same convention used by event-entropy-rate);
# other events use their kind. The raw phrase count produced by the
# incremental copy/insert parsing rule is normalized by the theoretical
# asymptotic upper bound n / log_b(n) (b = alphabet size) to stay in
# [0, 1] and remain comparable across traces of different
# lengths/alphabets. Unlike event-entropy-rate (first-order,
# previous-symbol-conditioned predictability), LZ76 complexity captures
# compressibility from arbitrary-length repeated substrings/motifs
# anywhere earlier in the sequence, without assuming a Markov model.
# Lower values indicate a highly compressible trace dominated by a few
# repeated motifs (likely stuck in a loop); values near 1 indicate a
# near-incompressible, structurally diverse trace.
lempel-ziv-trajectory-complexity:
name: Lempel-Ziv Trajectory Complexity
unit: ratio
direction: higher_is_better
min: 0.0
max: 1.0
script: |
const isRecord = value => value !== null && typeof value === "object" && !Array.isArray(value);

@github-actions github-actions Bot Aug 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/codebase-design] The isRecord helper and the candidates/event-extraction loop are verbatim copies from event-entropy-rate.md. Any future fix to the IR lookup strategy (e.g. adding a new key variant) must be applied to both files independently — a shotgun-surgery smell that will grow with each new event-sequence grader.

💡 Mitigation options
  1. If helpers (already used for helpers.clamp) is an extensible namespace, expose helpers.extractEventSequence(trace) there and call it from both graders.
  2. If not, a comment like // Canonical IR lookup — keep in sync with event-entropy-rate.md at minimum flags the coupling to future authors.

@copilot please address this.

const candidates = [
trace.trajectoryIR,
trace.trajectoryIr,
trace.ir,
isRecord(trace.agentOutput) ? trace.agentOutput.trajectoryIR : null,
isRecord(trace.agentOutput) ? trace.agentOutput.trajectoryIr : null,
isRecord(trace.agentOutput) ? trace.agentOutput.trajectory : null,
isRecord(trace.agentOutput) ? trace.agentOutput : null,
].filter(isRecord);

let events = [];
for (const candidate of candidates) {
if (Array.isArray(candidate.events)) {
events = candidate.events;
break;
}
}

// Build the ordered canonical event-symbol sequence: tool_call events
// are qualified by their ref for finer alphabet granularity, other
// events use their kind alone.
const ordered = events.map((event, position) => {
if (!isRecord(event) || typeof event.kind !== "string" || event.kind === "") return null;
const index = Number(event.index);
const order = Number.isFinite(index) ? index : position;
const symbol = event.kind === "tool_call" && typeof event.ref === "string" && event.ref !== "" ? `tool_call:${event.ref}` : event.kind;
return { symbol, order };
}).filter(entry => entry !== null).sort((a, b) => a.order - b.order);

const sequence = ordered.map(entry => entry.symbol);

@github-actions github-actions Bot Aug 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] The alphabetSize < 2 early return omits passed: null, while the n < 2 branch on line 52 includes it. If the grader framework uses passed: null as a signal for "not applicable" (distinct from a degenerate-but-valid score of 0), this inconsistency could cause routing issues downstream.

💡 Suggested fix 
if (alphabetSize < 2) {
  return { value: 0, unit: "ratio", passed: null, message: `not applicable: single-symbol alphabet (events=${n})` };
}

Aligning both early returns to use passed: null + message (rather than details) makes the "not applicable" intent unambiguous to the grader framework, consistent with event-entropy-rate.md line 52.

@copilot please address this.

const n = sequence.length;
if (n < 2) {
return { value: 0, unit: "ratio", passed: null, message: "not applicable: fewer than two events" };
}

const alphabetSize = new Set(sequence).size;
if (alphabetSize < 2) {
return { value: 0, unit: "ratio", details: `events=${n} alphabetSize=${alphabetSize}; LZ76 complexity is vacuously 0 with a single repeated symbol` };
}

// LZ76 incremental parsing (Kaspar & Schuster, 1987): scan the
// sequence, incrementally extending the current phrase while it can
// be found as a substring starting anywhere in [0, prefixLen);
// start a new phrase (increment complexity) each time the search
// pointer catches up to the end of the already-parsed prefix.
let complexity = 1;
let prefixLen = 1;
let subSeqLen = 1;
let maxSubSeqLen = 1;
let pointer = 0;
while (prefixLen + subSeqLen <= n) {
if (sequence[pointer + subSeqLen - 1] === sequence[prefixLen + subSeqLen - 1]) {
subSeqLen += 1;
} else {
maxSubSeqLen = Math.max(subSeqLen, maxSubSeqLen);
pointer += 1;
if (pointer === prefixLen) {
complexity += 1;
prefixLen += maxSubSeqLen;
pointer = 0;

@github-actions github-actions Bot Aug 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] The maxSubSeqLen tracking in the inner loop is a non-obvious LZ76 variant: when a mismatch occurs, subSeqLen resets to 1 but maxSubSeqLen preserves the best run, and the phrase advances by maxSubSeqLen. A comment citing which specific variant/reference this matches would help reviewers validate correctness.

💡 Why this matters: traced example for ["a","b","a","b"] 
n=4, complexity=1, prefixLen=1, subSeqLen=1, maxSubSeqLen=1, pointer=0

Step 1: seq[0]="a" vs seq[1]="b"? NO
  pointer→1 == prefixLen(1) → complexity=2, prefixLen=2, reset

Step 2: seq[0]="a" vs seq[2]="a"? YES → subSeqLen=2
Step 3: seq[1]="b" vs seq[3]="b"? YES → subSeqLen=3

prefixLen+subSeqLen=5 > 4 → exit
subSeqLen(3)!=1 → complexity=3
normalized = 3/4 = 0.75

Is 0.75 the expected value for period-2 ["a","b","a","b"]? Adding a short comment like // variant: Lempel-Ziv '76 copy/insert with maximal-copy phrase tracking and a link to the original paper would let the next reader verify this without reverse-engineering the loop.

@copilot please address this.

maxSubSeqLen = 1;

@github-actions github-actions Bot Aug 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This implementation overcounts LZ76 complexity on constant or single-phrase tails because if (subSeqLen !== 1) complexity += 1 blindly adds a phrase after the scan terminates, which makes a uniform sequence like ['a','a',...] report rawComplexity=2 instead of 1 and inflates the normalized score for the exact low-complexity traces this grader is supposed to catch.

💡 Why this blocks merge

The tail handling here is the fragile part of incremental LZ76 parsing: reaching the end of the scan with subSeqLen > 1 does not always mean a new phrase was discovered. In the degenerate repeated-symbol case, the loop exits only because the candidate extension hit the sequence boundary, but the parser never proved a new phrase boundary. Adding one unconditionally shifts the baseline upward and makes the metric materially wrong on the strongest "agent is stuck in a trivial loop" signal.

A minimal regression case is a trace where every canonical symbol is identical; this code currently yields complexity=2, even though there is only one phrase.

Please either switch to a known-correct LZ76 parsing implementation/test oracle or add explicit tests for edge cases like:

['a','a','a','a']           // should stay at the minimum complexity
['a','b','a','b','a','b']   // repeating motif
['a','b','c','d']           // maximal complexity for short unique trace

and only increment the final phrase count when the termination state actually represents a completed new phrase.

subSeqLen = 1;
} else {
subSeqLen = 1;
}
}
}
if (subSeqLen !== 1) complexity += 1;

// Normalize by the theoretical asymptotic upper bound n / log_b(n)
// (b = alphabet size) so the value stays in [0, 1] and remains

@github-actions github-actions Bot Aug 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comments claim that dividing by n / log_b(n) "stay[s] in [0, 1]", but the LZ76 asymptotic upper bound only holds for large n. For short sequences (as few as 7 symbols) the raw complexity can exceed this bound — e.g., ABAABBA yields c=5 vs upperBound≈2.49, a ratio of ~2.0. This means helpers.clamp is load-bearing and not merely defensive.

The code is correct because helpers.clamp does enforce [0, 1], but the comment misleads readers into thinking the formula alone guarantees the range. Please correct:

-      // Normalize by the theoretical asymptotic upper bound n / log_b(n)
-      // (b = alphabet size) so the value stays in [0, 1] and remains
-      // comparable across traces of different lengths/alphabets.
+      // Normalize by the theoretical asymptotic upper bound n / log_b(n)
+      // (b = alphabet size) to make the value comparable across traces of
+      // different lengths/alphabets. For short sequences the phrase count
+      // can exceed this asymptotic bound, so helpers.clamp enforces [0, 1].

The same correction applies to the YAML header comment that also says "to stay in [0, 1]".

@copilot please address this.

// comparable across traces of different lengths/alphabets.
const upperBound = n / (Math.log(n) / Math.log(alphabetSize));
const normalized = helpers.clamp(complexity / upperBound, 0, 1);

return {
value: normalized,
unit: "ratio",
details: `events=${n} alphabetSize=${alphabetSize} rawComplexity=${complexity} upperBound=${upperBound.toFixed(4)}`,
};
---

<!--
lempel-ziv-trajectory-complexity computes the normalized LZ76 (Kaspar &
Schuster) incremental-parsing complexity of the ordered canonical
event-symbol sequence built purely from the Trajectory IR's events[] (no
states, provenance, or objectives required). It measures whole-sequence

@github-actions github-actions Bot Aug 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] upperBound = n / log_b(n) is undefined when n = 1 (log of 1 = 0 → division by zero), and potentially very large when n is small (e.g. n=2, b=2 → upperBound=2/1=2, so normalized=1/2=0.5 which seems fine, but n=2, b=3 → log_3(2)≈0.63 → upperBound≈3.17 → complexity 2 / 3.17 → 0.63). The n < 2 guard prevents the n=1 case, but documenting or testing that helpers.clamp is the correct safety valve for small-n over-normalization would make the intention explicit.

💡 Suggestion

Add a comment above the upperBound line:

// For small n the asymptotic bound underestimates; clamp ensures value stays in [0,1].
const upperBound = n / (Math.log(n) / Math.log(alphabetSize));
const normalized = helpers.clamp(complexity / upperBound, 0, 1);

And consider adding a test vector for n=2 with both b=2 and b=alphabet-size to confirm the clamp behaves as intended.

@copilot please address this.

compressibility from arbitrary-length repeated substrings/motifs anywhere
earlier in the sequence, without assuming a Markov model — distinct from
event-entropy-rate, which only captures first-order (previous-symbol-
conditioned) statistical predictability, and from the recurrence-* family,
which requires canonical states and measures revisit density/structure
rather than whole-sequence compressibility.
-->
Loading