Add Twig and SCSS extractors - #2240
Open
amslezak wants to merge 2 commits into
Open
Conversation
Neither extension was in any FileType set, so Twig templates and Sass
stylesheets were silently skipped: on a Drupal theme that dropped the entire
component-composition and design-token layer from the graph.
Both are regex extractors in the shape of extractors/blade.py — no new
dependency, and routing them through FileType.CODE means they cost no LLM
tokens and yield EXTRACTED-confidence edges.
twig.py resolves references to real files rather than stubs:
- Drupal SDC (`{% include 'mytheme:card' %}`) via the nearest
`<provider>.info.yml` plus a components/ index
- plain paths and Symfony's `@Namespace/...` via the referring file's dir
and any enclosing templates/ root
- relations: includes, embeds, extends, imports_macro, uses_template,
defines_block, attaches_library
scss.py covers the module, mixin, and design-token graphs:
- @use/@forward/@import through Sass partial (`_name.scss`) and index
(`name/_index.scss`) conventions
- @mixin/@include keyed on the bare name so a namespaced use
(`@include mx.button-reset`) links to its definition
- `--token:` declarations and `var(--token)` references share one node, so
token-defining stylesheets connect to every consumer
Both strip comments before scanning, via a newline-preserving blank pass that
keeps reported line numbers accurate. Without it a component docblock showing
example usage ("Usage: {% include 'mytheme:card' %}") became a real dependency
and gave the component an edge to itself.
.twig/.scss/.sass are also added to _LANG_FAMILY_BY_EXT so their stubs cannot
be remapped onto same-labeled symbols in an unrelated language.
.css is deliberately out of scope: in a Sass project it is compiled output of
the .scss already indexed, so indexing both duplicates the stylesheet layer.
Verified against a 141-template / 109-stylesheet Drupal theme: 97% of Twig
references and 100% of Sass imports resolve to real files, no self-loops. The
unresolved remainder are base-theme templates outside the scanned corpus.
…iguation
A Single Directory Component keeps `card.twig` beside `card.scss`, so both
collapse to the same `card` file id and _disambiguate_colliding_node_ids salts
them apart. An `{% include %}` edge from a THIRD template carries neither
salt's source_key, so the (target, edge_source_key) lookup missed and the edge
dangled on the retired `card` id.
That is the Graphify-Labs#1475 C/ObjC header failure exactly — `foo.h` beside `foo.c`, with
the import edge from a third file. Mirror its fix: repoint template-composition
edges to the .twig variant, since an include always targeted the template.
Also honor the target_file stamp for those relations. It was gated to
imports/imports_from/re_exports, so a stamped `includes` edge fell back to the
importer's own source_key.
On a 141-template Drupal theme this takes dangling composition edges from
198/234 to 0/234, and embeds from 14/14 to 0/14. The extractors emit the edge
only (no stub node) for a resolved reference, matching the documented Graphify-Labs#2195
pattern, so no ghost node duplicates the real file node either.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds extractors for Twig templates and SCSS/Sass stylesheets. Neither extension appeared in any
FileTypeset, so both were skipped at detection —rg twigover the package returns nothing today. On a Drupal or Symfony project that means the entire template-composition layer is missing from the graph, and on any Sass project the stylesheet module graph is too.Both extractors follow
extractors/blade.py: pure regex, no new dependency. Routing them throughFileType.CODErather than the LLM document path means they cost no tokens and produceEXTRACTED-confidence edges with precise relation names.Two commits: the extractors, then a resolution fix they exposed.
1. The extractors
graphify/extractors/twig.pyresolves references to real files rather than leaving stubs, so templates link to templates:{% include 'mytheme:card' %}, resolved via the nearest ancestor<provider>.info.ymlplus an index ofcomponents/**/<name>/<name>.twig.{% extends 'layout.html.twig' %}and@Namespace/path.html.twig, resolved against the referring file's directory and any enclosingtemplates/root.Relations:
includes,embeds,extends,imports_macro,uses_template,defines_block,attaches_library. Covers the{% tag %}form, the{{ include('x') }}function form, and{% from 'x' import y %}.graphify/extractors/scss.pycovers three graphs:@use/@forward/@importthrough Sass's partial (_name.scss) and index (name/_index.scss) conventions. Handles@import 'a', 'b'lists; leavessass:matha stub.@mixin/@includekeyed on the bare name, so a namespaced use (@include mx.button-reset) links to its definition instead of orphaning.--token:declarations andvar(--token)references.Both follow the #2195 pattern: a resolved reference emits the edge only, stamped with
target_file, and mints no stub node — a second node carrying the same id would be a ghost duplicate of the real file node.Comment stripping
Both strip comments first, via a newline-preserving blank pass (
_blank_spansinbase.py) so reported line numbers stay accurate.This is load-bearing. Component docblocks routinely carry a usage example:
Read naively that gives the card component an
includesedge to itself — a self-loop, whichdiagnose_extractionflags. Same for a commented-out// @use 'ghost';.2. Keeping composition edges alive through id disambiguation
Validating against a real theme surfaced a resolution bug that isn't Twig-specific.
A Single Directory Component keeps
card.twigbesidecard.scss. Both collapse to the samecardfile id, so_disambiguate_colliding_node_idssalts them apart by path. An{% include %}edge from a third template carries neither salt'ssource_key, so the(target, edge_source_key)lookup misses and the edge dangles on the now-retiredcardid.That is the #1475 C/ObjC header failure exactly —
foo.hbesidefoo.c, with the import arriving from a third file. This mirrors that fix: repoint template-composition edges to the.twigvariant, since an include always targeted the template.It also honors the
target_filestamp for those relations. The gate was("imports", "imports_from", "re_exports"), so a stampedincludesedge fell back to the importer's ownsource_key.Measured on a 141-template / 109-stylesheet Drupal theme:
includesdanglingembedsdanglingextendsdanglingThe only unresolved Twig references left are base-theme templates outside the scanned directory, which correctly stay stubs.
Also
.twig/.scss/.sassadded to_LANG_FAMILY_BY_EXT. With no named family_lang_familyreturnsNone, which lets the stub-remap pass bind e.g. an SCSS mixin stub to a same-labeled function in an unrelated language..cssis deliberately out of scope. In a Sass project the CSS files are compiled output of the.scssalready indexed, so adding it duplicates the stylesheet layer. Worth a separate discussion for hand-written-CSS projects.Testing
tests/test_twig.pyandtests/test_scss.py— 44 new tests, plus classification tests intests/test_detect.py. Both Twig dialects are covered (Drupal SDC and plain-path/Symfony), so this isn't a Drupal-only patch. Tests pin the behaviors that were actually wrong during development: the docblock self-include, single-line--token:declarations, thetarget_filestamp, and existence-gating so an unresolvable reference still dangles.Full suite: 3,757 passed, no new failures. (15 pre-existing failures in
test_skillgen.py/test_ollama_retry_cap.pyreproduce identically on a clean0.9.28checkout, untouched by this branch.)Known limitation
Design-token nodes don't collapse into one shared node —
--spacing-05referenced from 59 stylesheets becomes 59 per-file nodes, because disambiguation salts them apart like any same-named symbol. Consumers are still findable by label, butdefines_tokennever reachesuses_tokenacross files. Thetype: "module"anchor exemption (#1327) looks like the right mechanism, but that field doesn't survive the AST node path, so I left it out rather than ship something inert. Happy to follow up if you'd like it fixed.