Automatic asm.js dictionary extraction and clang toolchain update - #203
Merged
Conversation
…y from the build Build the wasm with the latest upstream emscripten emcc (modern clang/LLVM) instead of the WASI SDK, and keep the fast fastcomp asm.js build on its own pinned emsdk. The asm.js dictionary offset and static base are now extracted from the fastcomp output at build time rather than hand-maintained.
1 task
guybedford
added a commit
that referenced
this pull request
Jul 2, 2026
The emsdk toolchain update in #203 dropped the growable memory the old fastcomp build emitted: the wasm memory was declared with maximum == initial (258 pages / ~16MiB), so the unconditional memory.grow() in parse() throws "Maximum memory size exceeded" for any source over ~4M chars (stored as 4 bytes per char). Add -s ALLOW_MEMORY_GROWTH=1 to restore growable memory, and a 5MB large source regression test. Fixes #214
guybedford
added a commit
that referenced
this pull request
Jul 2, 2026
The emsdk toolchain update in #203 dropped the growable memory the old fastcomp build emitted: the wasm memory was declared with maximum == initial (258 pages / ~16MiB), so the unconditional memory.grow() in parse() throws "Maximum memory size exceeded" for any source over ~4M chars (stored as 4 bytes per char). Add -s ALLOW_MEMORY_GROWTH=1 to restore growable memory, and a 5MB large source regression test. Fixes #214
mergify Bot
added a commit
to ArcadeData/arcadedb
that referenced
this pull request
Jul 5, 2026
…2.3.0 in /studio [skip ci] Bumps [es-module-lexer](https://github.com/guybedford/es-module-lexer) from 2.1.0 to 2.3.0. Release notes *Sourced from [es-module-lexer's releases](https://github.com/guybedford/es-module-lexer/releases).* > 2.2.0 > ----- > > What's Changed > -------------- > > * build(deps-dev): update dev dependencies to latest by [`@BridgeAR`](https://github.com/BridgeAR) in [guybedford/es-module-lexer#200](https://redirect.github.com/guybedford/es-module-lexer/pull/200) > * fix: report n for no-substitution template dynamic imports by [`@BridgeAR`](https://github.com/BridgeAR) in [guybedford/es-module-lexer#201](https://redirect.github.com/guybedford/es-module-lexer/pull/201) > * Automatic asm.js dictionary extraction and clang toolchain update by [`@guybedford`](https://github.com/guybedford) in [guybedford/es-module-lexer#203](https://redirect.github.com/guybedford/es-module-lexer/pull/203) > * fix: don't treat a method named `import` with 2+ args as a dynamic import by [`@soberm`](https://github.com/soberm) in [guybedford/es-module-lexer#207](https://redirect.github.com/guybedford/es-module-lexer/pull/207) > * fix: detect every binding in export var/let/const declarations by [`@BridgeAR`](https://github.com/BridgeAR) in [guybedford/es-module-lexer#199](https://redirect.github.com/guybedford/es-module-lexer/pull/199) > * feat: expose export statement start position as ss by [`@BridgeAR`](https://github.com/BridgeAR) in [guybedford/es-module-lexer#206](https://redirect.github.com/guybedford/es-module-lexer/pull/206) > > New Contributors > ---------------- > > * [`@BridgeAR`](https://github.com/BridgeAR) made their first contribution in [guybedford/es-module-lexer#200](https://redirect.github.com/guybedford/es-module-lexer/pull/200) > * [`@soberm`](https://github.com/soberm) made their first contribution in [guybedford/es-module-lexer#207](https://redirect.github.com/guybedford/es-module-lexer/pull/207) > > **Full Changelog**: <guybedford/es-module-lexer@2.1.0...2.2.0> Commits * [`dbac1c3`](guybedford/es-module-lexer@dbac1c3) 2.3.0 * [`f3c972d`](guybedford/es-module-lexer@f3c972d) fix: allow wasm memory growth in the minimal build, document it in the readme... * [`e9b611e`](guybedford/es-module-lexer@e9b611e) fix: allow wasm memory growth for sources over ~4MB ([#217](https://redirect.github.com/guybedford/es-module-lexer/issues/217)) * [`6c90425`](guybedford/es-module-lexer@6c90425) feat: add minimal build for es-module-shims (es-module-lexer/minimal) ([#211](https://redirect.github.com/guybedford/es-module-lexer/issues/211)) * [`70010fb`](guybedford/es-module-lexer@70010fb) docs: correct stale limitations and gzip size in README ([#212](https://redirect.github.com/guybedford/es-module-lexer/issues/212)) * [`1f19494`](guybedford/es-module-lexer@1f19494) 2.2.0 * [`d61e97a`](guybedford/es-module-lexer@d61e97a) feat: expose export statement start position as ss ([#206](https://redirect.github.com/guybedford/es-module-lexer/issues/206)) * [`064d0db`](guybedford/es-module-lexer@064d0db) fix: detect every binding in export var/let/const declarations ([#199](https://redirect.github.com/guybedford/es-module-lexer/issues/199)) * [`99b5226`](guybedford/es-module-lexer@99b5226) fix: don't treat a method named `import` with 2+ args as a dynamic import ([#207](https://redirect.github.com/guybedford/es-module-lexer/issues/207)) * [`fa1ade6`](guybedford/es-module-lexer@fa1ade6) build: update to upstream emsdk toolchain and derive asm.js dictionary from t... * Additional commits viewable in [compare view](guybedford/es-module-lexer@2.1.0...2.3.0) [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This makes the asm.js build dictionary-independent, and updates the build to the latest clang toolchain.
The asm.js build previously required manual dictionary construction: the keyword dictionary, its write offset into the heap, and the static base were reverse-engineered by hand and hardcoded in the wrapper (the "manual asm dictionary construction" note in lexer.c). These silently desync whenever the static memory layout shifts (for example under different optimization flags), so editing the lexer was error-prone. They are now derived from the compiled output at build time, making lexer.c the single source of truth for the keywords — adding or changing a keyword needs no downstream hand-editing.
-Oz -flto -s SUPPORT_LONGJMP=0)AGGRESSIVE_VARIABLE_ELIMINATIONenabled on the asm.js buildFootprint vs main:
dist/lexer.js: 6,189 -> 5,994B Brotli (-195B)dist/lexer.asm.js: ~flat (5,930 -> 5,935B)asm.js runtime performance is unchanged. All 132 tests pass for both the wasm and asm builds.