Skip to content

Add Twig and SCSS extractors - #2240

Open
amslezak wants to merge 2 commits into
Graphify-Labs:v8from
amslezak:feat/twig-scss-extractors
Open

Add Twig and SCSS extractors#2240
amslezak wants to merge 2 commits into
Graphify-Labs:v8from
amslezak:feat/twig-scss-extractors

Conversation

@amslezak

Copy link
Copy Markdown

What

Adds extractors for Twig templates and SCSS/Sass stylesheets. Neither extension appeared in any FileType set, so both were skipped at detection — rg twig over the package returns nothing today. On a Drupal or Symfony project that means the entire template-composition layer is missing from the graph, and on any Sass project the stylesheet module graph is too.

Both extractors follow extractors/blade.py: pure regex, no new dependency. Routing them through FileType.CODE rather than the LLM document path means they cost no tokens and produce EXTRACTED-confidence edges with precise relation names.

Two commits: the extractors, then a resolution fix they exposed.

1. The extractors

graphify/extractors/twig.py resolves references to real files rather than leaving stubs, so templates link to templates:

  • Drupal SDC{% include 'mytheme:card' %}, resolved via the nearest ancestor <provider>.info.yml plus an index of components/**/<name>/<name>.twig.
  • Plain paths and Symfony{% extends 'layout.html.twig' %} and @Namespace/path.html.twig, resolved against the referring file's directory and any enclosing templates/ root.

Relations: includes, embeds, extends, imports_macro, uses_template, defines_block, attaches_library. Covers the {% tag %} form, the {{ include('x') }} function form, and {% from 'x' import y %}.

graphify/extractors/scss.py covers three graphs:

  • Modules@use / @forward / @import through Sass's partial (_name.scss) and index (name/_index.scss) conventions. Handles @import 'a', 'b' lists; leaves sass:math a stub.
  • Mixins@mixin / @include keyed on the bare name, so a namespaced use (@include mx.button-reset) links to its definition instead of orphaning.
  • Design tokens--token: declarations and var(--token) references.

Both follow the #2195 pattern: a resolved reference emits the edge only, stamped with target_file, and mints no stub node — a second node carrying the same id would be a ghost duplicate of the real file node.

Comment stripping

Both strip comments first, via a newline-preserving blank pass (_blank_spans in base.py) so reported line numbers stay accurate.

This is load-bearing. Component docblocks routinely carry a usage example:

{#
 * Usage:
 *   {% include 'mytheme:card' with { title: 'Hello' } %}
 #}

Read naively that gives the card component an includes edge to itself — a self-loop, which diagnose_extraction flags. Same for a commented-out // @use 'ghost';.

2. Keeping composition edges alive through id disambiguation

Validating against a real theme surfaced a resolution bug that isn't Twig-specific.

A Single Directory Component keeps card.twig beside card.scss. Both collapse to the same card file id, so _disambiguate_colliding_node_ids salts them apart by path. An {% include %} edge from a third template carries neither salt's source_key, so the (target, edge_source_key) lookup misses and the edge dangles on the now-retired card id.

That is the #1475 C/ObjC header failure exactly — foo.h beside foo.c, with the import arriving from a third file. This mirrors that fix: repoint template-composition edges to the .twig variant, since an include always targeted the template.

It also honors the target_file stamp for those relations. The gate was ("imports", "imports_from", "re_exports"), so a stamped includes edge fell back to the importer's own source_key.

Measured on a 141-template / 109-stylesheet Drupal theme:

before after
includes dangling 198 / 234 0 / 234
embeds dangling 14 / 14 0 / 14
extends dangling 1 / 1 0 / 1
Sass imports resolved 84 / 84
self-loop edges 0

The only unresolved Twig references left are base-theme templates outside the scanned directory, which correctly stay stubs.

Also

  • .twig / .scss / .sass added to _LANG_FAMILY_BY_EXT. With no named family _lang_family returns None, which lets the stub-remap pass bind e.g. an SCSS mixin stub to a same-labeled function in an unrelated language.
  • .css is deliberately out of scope. In a Sass project the CSS files are compiled output of the .scss already indexed, so adding it duplicates the stylesheet layer. Worth a separate discussion for hand-written-CSS projects.

Testing

tests/test_twig.py and tests/test_scss.py — 44 new tests, plus classification tests in tests/test_detect.py. Both Twig dialects are covered (Drupal SDC and plain-path/Symfony), so this isn't a Drupal-only patch. Tests pin the behaviors that were actually wrong during development: the docblock self-include, single-line --token: declarations, the target_file stamp, and existence-gating so an unresolvable reference still dangles.

Full suite: 3,757 passed, no new failures. (15 pre-existing failures in test_skillgen.py / test_ollama_retry_cap.py reproduce identically on a clean 0.9.28 checkout, untouched by this branch.)

Known limitation

Design-token nodes don't collapse into one shared node — --spacing-05 referenced from 59 stylesheets becomes 59 per-file nodes, because disambiguation salts them apart like any same-named symbol. Consumers are still findable by label, but defines_token never reaches uses_token across files. The type: "module" anchor exemption (#1327) looks like the right mechanism, but that field doesn't survive the AST node path, so I left it out rather than ship something inert. Happy to follow up if you'd like it fixed.

amslezak added 2 commits July 27, 2026 12:55
Neither extension was in any FileType set, so Twig templates and Sass
stylesheets were silently skipped: on a Drupal theme that dropped the entire
component-composition and design-token layer from the graph.

Both are regex extractors in the shape of extractors/blade.py — no new
dependency, and routing them through FileType.CODE means they cost no LLM
tokens and yield EXTRACTED-confidence edges.

twig.py resolves references to real files rather than stubs:
- Drupal SDC (`{% include 'mytheme:card' %}`) via the nearest
  `<provider>.info.yml` plus a components/ index
- plain paths and Symfony's `@Namespace/...` via the referring file's dir
  and any enclosing templates/ root
- relations: includes, embeds, extends, imports_macro, uses_template,
  defines_block, attaches_library

scss.py covers the module, mixin, and design-token graphs:
- @use/@forward/@import through Sass partial (`_name.scss`) and index
  (`name/_index.scss`) conventions
- @mixin/@include keyed on the bare name so a namespaced use
  (`@include mx.button-reset`) links to its definition
- `--token:` declarations and `var(--token)` references share one node, so
  token-defining stylesheets connect to every consumer

Both strip comments before scanning, via a newline-preserving blank pass that
keeps reported line numbers accurate. Without it a component docblock showing
example usage ("Usage: {% include 'mytheme:card' %}") became a real dependency
and gave the component an edge to itself.

.twig/.scss/.sass are also added to _LANG_FAMILY_BY_EXT so their stubs cannot
be remapped onto same-labeled symbols in an unrelated language.

.css is deliberately out of scope: in a Sass project it is compiled output of
the .scss already indexed, so indexing both duplicates the stylesheet layer.

Verified against a 141-template / 109-stylesheet Drupal theme: 97% of Twig
references and 100% of Sass imports resolve to real files, no self-loops. The
unresolved remainder are base-theme templates outside the scanned corpus.
…iguation

A Single Directory Component keeps `card.twig` beside `card.scss`, so both
collapse to the same `card` file id and _disambiguate_colliding_node_ids salts
them apart. An `{% include %}` edge from a THIRD template carries neither
salt's source_key, so the (target, edge_source_key) lookup missed and the edge
dangled on the retired `card` id.

That is the Graphify-Labs#1475 C/ObjC header failure exactly — `foo.h` beside `foo.c`, with
the import edge from a third file. Mirror its fix: repoint template-composition
edges to the .twig variant, since an include always targeted the template.

Also honor the target_file stamp for those relations. It was gated to
imports/imports_from/re_exports, so a stamped `includes` edge fell back to the
importer's own source_key.

On a 141-template Drupal theme this takes dangling composition edges from
198/234 to 0/234, and embeds from 14/14 to 0/14. The extractors emit the edge
only (no stub node) for a resolved reference, matching the documented Graphify-Labs#2195
pattern, so no ghost node duplicates the real file node either.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant