diff --git a/docs/document_alignment_guide.md b/docs/document_alignment_guide.md index 50e8e9b5c5..6ea4ad7c8f 100644 --- a/docs/document_alignment_guide.md +++ b/docs/document_alignment_guide.md @@ -36,7 +36,7 @@ either misplaced or the thesis needs updating — don't let a page just drift. |---|---|---|---| | 0 — Entry hub | `README.md` | First 90 seconds. Every claim here must be evidence-gated. | `how_to_maintain_the_readme.md`, `readme_evidence_roadmap.md` | | 1 — Deep hub | `docs/wiki/index.md` | Full doc-site index, routes by role (security architect, legacy team, systems engineer). | This doc | -| 2 — Foundation & claims | `docs/wiki/01-*`, `03-*` | The thesis, spelled out: the blAST paradigm, the 10 numbered "Claims" pages, future outlook. | This doc (tone gap noted below) | +| 2 — Foundation & claims | `docs/wiki/01-*`, `03-*` | The thesis, spelled out: the blAST paradigm, the 11 numbered "Claims" pages, future outlook. | This doc (tone gap noted below) | | 2 — Pipeline & math | `docs/wiki/02-*`, `06-*`, `07-*`, `08-*` | Stage-by-stage pipeline docs, per-signal risk-equation docs. One page per pipeline stage / equation, mirrors `gitgalaxy/core/README.md` and `gitgalaxy/metrics/`. | Should match the code at the path it documents — no separate rule doc yet (candidate follow-up, see below) | | 3 — Evidence artifacts | `docs/language_status/`, `docs/self_scan/` (chart/CSV), `language-crucible`, `gitgalaxy-raw-output` (external repos) | What backs the claims — real measurements, not prose. | `.claude/skills/language-status/SKILL.md` | | 4 — Applied spokes | `docs/wiki/04-*`/`05-*` (security tools, legacy tools), `cookbook/`, `agents/`, `LLM-reports/`, `museum-of-code/` | Domain-specific application of the thesis; task-oriented, not proof-oriented. | This doc | @@ -52,6 +52,7 @@ The load-bearing claims in `README.md`, and where a reader lands if they click t | Correctness on real, uncompilable production code | `#proof-not-just-claims` item 2 | — | `language-crucible`, `tests/golden_master_audit.json` | | Runs unmodified at real-world scale | `#proof-not-just-claims` item 3 | [`museum-of-code/`](wiki/museum-of-code/index.md) (per-repo teardowns), [`LLM-reports/`](wiki/LLM-reports/index.md) | `gitgalaxy-raw-output` | | Measured extraction accuracy vs. Tree-sitter ground truth | `#proof-not-just-claims` item 4 | [`docs/language_status/`](language_status/README.md) | `docs/self_scan/tree_sitter_accuracy_chart.svg` + `_history.csv`, `tests/tree_sitter_accuracy_baseline_*.json` | +| Multi-signal language detection with provenance, refusal, and repo-context | `#one-graph-not-five-separate-tools` | [`03-11-claim-11-multi-signal-detection`](wiki/03-11-claim-11-multi-signal-detection.md), [`02-05-language-lens`](wiki/02-05-language-lens.md) | `tests/detection_accuracy_baseline.json`, `language-crucible` (contested subset) | | Scan speed / scales near-linearly | "What Pain Point Does This Solve?" | [`03-01-claim-1-search-strategies`](wiki/03-01-claim-1-search-strategies.md) | `gitgalaxy-raw-output`'s speed telemetry | | One graph feeds many tools (security, legacy, SBOM, AI guardrails) | "One Graph, Not Five Separate Tools", "Enterprise Codebase Tools & Use Cases" | `docs/wiki/04-*` (security), `05-*` (legacy), `cookbook/*` | Per-tool benchmark bullets in `README.md` itself | | Risk scores are prioritization signals, not verdicts | "What GitGalaxy Finds — and What It Doesn't Claim" | [`08-01-methodology`](wiki/08-01-methodology.md) onward | — (methodology) | diff --git a/docs/wiki/02-05-language-lens.md b/docs/wiki/02-05-language-lens.md index 63e1b3a34c..a3f460105b 100644 --- a/docs/wiki/02-05-language-lens.md +++ b/docs/wiki/02-05-language-lens.md @@ -50,4 +50,6 @@ Expanding the transition marker registry to better handle nested JSX/TSX and tem ## Related Components - [Guidestar Protocol](02-04-guidestar-protocol.md) - [The Prism](02-07-the-prism.md) -(index.md)** \ No newline at end of file +- [Claim 11: Multi-Signal Language Detection](03-11-claim-11-multi-signal-detection.md) — the argued case for this engine (why extension is an assertion, not evidence), with the measured accuracy figures. + +**[⬅️ Back to Master Index](index.md)** \ No newline at end of file diff --git a/docs/wiki/03-11-claim-11-multi-signal-detection.md b/docs/wiki/03-11-claim-11-multi-signal-detection.md new file mode 100644 index 0000000000..90b25c0e33 --- /dev/null +++ b/docs/wiki/03-11-claim-11-multi-signal-detection.md @@ -0,0 +1,76 @@ +# Claim 11: Multi-Signal Language Detection (Inference With Provenance, Not a Waterfall) + +Most language detectors answer one question — *"what language is this file?"* — and return one thing: a label. GitGalaxy answers a different question — *"what language is this file, how do we know, and how much should you trust that?"* — and returns three things: a language, a **trust tier**, and a human-readable **proof of how it was reached**. + +The premise is that a file extension is an **assertion, not evidence**. `payload.json` can be a shell script; `notes.txt` can be a PowerShell payload with a `.txt` coat on. Treating the extension as ground truth is how a scanner gets lied to. So detection here is a multi-signal inference that weighs several independent, zero-dependency signals against each other and attaches an explicit confidence to the verdict — never a single waterfall that falls through to a bare guess. + +This page argues the design. For the mechanical tier-by-tier reference, see **[Language Lens](02-05-language-lens.md)**. + +--- + +## 1. The signals actually combined + +All of these are computed with **no trained model and no sample corpus** — pure structure, resolvable on an air-gapped box: + +* **Exact filename anchors** — `Makefile`, `Dockerfile`, `setup.py`, `SConstruct` are their language regardless of extension. +* **Extension**, with multi-suffix unwrapping for wrapper extensions — `script.sh.template` resolves through `.template` to `.sh`, but only when the inner suffix is a known safe wrapper (so `malware.exe.txt` is *not* unwrapped into an executable). +* **Shebang**, matched on the interpreter basename as a token — never an unanchored substring (the #3116 fix: a canonical `#!/usr/bin/tclsh` is no longer defeated by, nor confused with, an unrelated line that merely contains those characters). +* **Prose anchors** — README / LICENSE / CONTRIBUTING-family files, with affix matching. +* **Sibling anchors** — `foo.h` resolves to C when `foo.c` is in the directory tally, C++ when `foo.cpp` is, Objective-C when `foo.m` is. +* **Ecosystem gravity** — a directory-local census combined with per-language discriminators and disqualifiers, gated on `ECOSYSTEM_DOMINANCE_MIN` (0.70). For a *same-extension* collision — where counting extensions is information-free because both rivals share the extension — gravity now weighs what the neighbours **actually classified as by their own content**, not merely their filenames (the #3137 sibling-classification vote). +* **`internal_discriminator` content regexes** — consulted *only* for registry-declared collision extensions, never as global scanners (an Objective-C `@implementation` pattern must not be free to hijack an unrelated `.m` MATLAB file). +* **A full lexical scan** that scores the file against each candidate language's own structural-signature rules. +* **A discovery funnel** for extensionless files that isolates the comment family first, then requires a 1.5× margin over the runner-up before it will name a language. +* **Caller-supplied context priors** from upstream metadata. + +--- + +## 2. The three genuinely differentiated properties + +This is the honest core of the claim — the part with no obvious counterpart in Linguist, enry, guesslang, Pygments, or libmagic. + +* **Agreement-based tiering with provenance.** Two independent signals concurring (extension *and* shebang, say) lock **Tier 0** at 0.999; a single indicator lands **Tier 2** at 0.91. Every result carries a numeric `lock_tier` and a human-readable `source_proof` string — `"Absolute Consensus (Ext + Shebang)"`, `"Sibling Anchor (.c)"`, `"Collision Resolved (.py -> embedded_python)"`. The engine reports not just the answer but *how it was reached and how much to trust it*. Mainstream detectors return a label and keep their reasoning to themselves. + +* **Identity conflict as a refusal — and as a security signal.** When the extension and the shebang name two different languages, the file is **not** resolved to a best guess. It drops to **Tier 5, "Absolute Distrust"**: `undeterminable`, intensity `0.0`, with an `Identity Masking` anomaly flag raised for the security engine. The contradiction *is* the finding — a `.txt` that re-execs a shell is more interesting *because* it lies about itself than any confident label could ever be. A generic-shell launcher that re-execs an interpreter its own extension already claims (a portable `.tcl` trampoline) is explicitly cleared, so the refusal fires on masquerade, not on legitimate bootstrapping. + +* **Repo-context inference.** Gravity and sibling anchors classify a file using its **neighbours** — something a stateless, file-by-file classifier structurally cannot express. This is what makes the mainframe family tractable: an EQU-only copybook is HLASM *because it sits beside JCL and COBOL*, and a telemetry `.py` with no hardware imports is MicroPython *because every resolvable file around it resolved to MicroPython by content*. A per-file oracle has no vocabulary for "because of where it lives." + +--- + +## 3. Where it is convergent, not novel + +Stating this plainly makes the claim *stronger*, not weaker. + +The collision-registry-plus-content-discriminator mechanism is, functionally, what GitHub Linguist does in `heuristics.yml`. The per-language lexical self-scoring is, architecturally, what Pygments does with `analyse_text()`. We did not invent disambiguation-by-content or scoring-by-structural-fingerprint. + +Arriving independently at the same disambiguation structure as the tool calibrated against millions of repositories is a **validation** of the design — not an embarrassment, and not something to dress up as invention. A writeup that claimed novelty for those two parts would be falsifiable in five minutes by anyone who knows the field, and would discredit the three real claims above by association. + +The honest framing: *GitGalaxy independently arrived at the field's disambiguation architecture — with no dependencies and no training data — and then added provenance tiering, identity-conflict refusal, and repo-context inference on top of it.* + +--- + +## 4. Honest limits + +Mirroring Claim 10's own §3, because that section is what makes it credible. + +**The lexical scan's scoring constants are hand-tuned and uncalibrated.** In `_tier_3_lexical_scan` the score is assembled from: per-rule-hit weights; a comment-delimiter bonus of `+= 15.0` awarded on a plain substring test (`if d in content`); two `1.25×` boosts (one for the gravity winner, one for the extension-tier claimed language); a hardcoded `0.4×` handicap for exactly three broad-ruleset languages (`if lid in ("abap", "fortran", "cobol")`); log-normalisation by lines of code (`raw_score / math.log1p(loc)`); and a final confidence derived by dividing the top signal by 50 (`min(top_signal / 50.0, 1.0)`). These numbers work on the corpus we have. None of them is calibrated against a probabilistic model, and that is the next real piece of work here, not a solved problem. + +**The design catches its own defects — and this section is the receipt.** The [#3117 detection-accuracy harness](02-05-language-lens.md) exists precisely so that a stably-wrong classification cannot hide behind green golden-master tests. On its first runs it surfaced, and the engine has since closed, four real defects: an EQU-only copybook misclassified via gravity (#3110); canonical shebangs forced to a false Tier 5 conflict by unanchored substring matching (#3116); silent tie-breaking by registry order on contested extensions (#3118); and gravity's structural blindness to same-extension collisions, which resolved by pinning to a discriminator self-reference until the sibling-classification vote replaced it (#3132 / #3137). A detector whose own harness finds real bugs and whose maintainers fix rather than relabel them is the honest kind. + +--- + +## 5. The number + +A claims page that asserts architectural superiority with no measured accuracy is the weakest possible form of this document. So here is the measurement, from the #3117 harness against the pinned 51-language `language-crucible` corpus (3,531 files): + +* **Determinable accuracy: 3,231 / 3,231 = 1.0** — every file with a single defensible language got that language. +* **Ambiguous rejection: 1 / 1 = 1.0** — every file with *no* single defensible language was declined (`undeterminable`), not guessed. This objective is scored *inverted*: a refusal passes, a confident verdict fails. Driving one objective up by guessing harder drives the other down, so the two cannot be gamed the same way. +* **Confusion matrix: empty.** + +The honest caveat travels with the figure. Most of the corpus is auto-labelled *by extension*, and extension is also the detector's primary signal — so on that subset the oracle and the system under test share an input, and a high score there is partly tautological. The number that actually measures disambiguation is the **hand-labelled contested subset** (files on shared extensions, where content and context must decide): **509 / 509 = 1.0**. And 1.0 on a finite, pinned corpus is a **generalisation estimate**, not a proof of universal perfection — its value is as evidence that the two failure modes are both driven to target on independent ground truth, not as a guarantee about all code everywhere. + +--- + +**See also:** [Language Lens](02-05-language-lens.md) — the mechanical tier-by-tier reference for the engine this claim argues for. + +**[⬅️ Back to Master Index](index.md)** diff --git a/docs/wiki/index.md b/docs/wiki/index.md index 1504cd3433..02fdc03d31 100755 --- a/docs/wiki/index.md +++ b/docs/wiki/index.md @@ -29,6 +29,7 @@ If you are tasked with breaking apart 40-year-old IBM monoliths without using an If you want to understand the mathematics driving the engine and the empirical proofs that validate our AST-free approach: * 🔬 [The blAST Paradigm (Heuristics vs. ASTs vs. LLMs)](01-03-the-blast-paradigm.md) * ⚖️ [Claim 10: The Heuristic vs. AST Paradigm](03-10-claim-10-ast-vs-heuristic-parsing.md) +* 🔎 [Claim 11: Multi-Signal Language Detection (Inference With Provenance)](03-11-claim-11-multi-signal-detection.md) * 🛡️ [Claim 8: Empirical Validation of AST-Free Parsing (The Gauntlets)](03-08-claim-8-empirical-validation-of-ast-free-parsing.md) * 📐 [The 13-Point Structural Surface Profile](08-01-methodology.md) — formerly "Risk Exposure"; see [the vector reference & validation record](../vectors.md) for why the `risk_` framing was retired @@ -74,6 +75,7 @@ This is a curated subset for first-time visitors, not the complete page list -- * [03-08: Claim 8 - AST-Free Empirical Validation](03-08-claim-8-empirical-validation-of-ast-free-parsing.md) * [03-09: Claim 9 - Universal Logic Extraction](03-09-claim-9-universal-logic-extraction.md) * [03-10: Claim 10 - AST vs Heuristic Paradigm](03-10-claim-10-ast-vs-heuristic-parsing.md) +* [03-11: Claim 11 - Multi-Signal Language Detection](03-11-claim-11-multi-signal-detection.md) * [03-20: Future Outlooks](03-20-future-outlooks.md) ### Pipeline & Architecture diff --git a/mkdocs.yml b/mkdocs.yml index 200a004b15..8544eb31bb 100755 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -329,6 +329,7 @@ nav: - 'Claim 8 (AST-Free Empirical Validation)': "03-08-claim-8-empirical-validation-of-ast-free-parsing.md" - 'Claim 9 (Universal Logic Extraction)': "03-09-claim-9-universal-logic-extraction.md" - 'Claim 10 (AST vs Heuristic Paradigm)': "03-10-claim-10-ast-vs-heuristic-parsing.md" + - 'Claim 11 (Multi-Signal Language Detection)': "03-11-claim-11-multi-signal-detection.md" - 'Future Outlooks': "03-20-future-outlooks.md" - '4. Security Tools & Spokes': - 'Competitive Landscape': "04-00-security_landscape.md"