Remap the legacy Gemma 1 hidden_act in the config post-init by PCfVW · Pull Request #49084 · huggingface/transformers

PCfVW · 2026-09-24T12:48:17Z

The Gemma 1.0 checkpoints ship hidden_act="gelu", which resolves to the exact erf GELU, but they were trained with the tanh approximation. GemmaMLP used to correct this by reading hidden_activation; #35235 dropped that field and left the legacy value in force.

As @vasqu suggested on the issue, the remap goes in GemmaConfig.__post_init__ rather than in model init. It therefore runs after from_dict, which is the path every Hub config takes, and it means save_pretrained and anything else reading the config see the corrected value too, not just GemmaMLP.

Verified on google/gemma-2b: with this patch the logits are torch.equal to a run forced to gelu_pytorch_tanh, on CPU/F32 and CUDA/BF16. Without it, the top-10 max abs-diff is 1.978e-2 in F32, and BF16 reorders 7 of 30 top-10 positions.

Tests cover the legacy value, the from_dict path, the save_pretrained round-trip, the default, and deliberate values (gelu_new, silu, relu) left alone. The four that assert the fix fail without it.

The Gemma 1.0 checkpoints ship `hidden_act="gelu"`, which resolves to the exact erf GELU, but they were trained with the tanh approximation. `GemmaMLP` used to correct this by reading `hidden_activation`; huggingface#35235 dropped that field and left the legacy value in force, silently. Remapping in `GemmaConfig.__post_init__` rather than in the model runs after `from_dict`, so it covers configs loaded from the Hub, and it means `save_pretrained` and anything else reading the config see the corrected value too, rather than only `GemmaMLP`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

PCfVW · 2026-09-24T12:53:00Z

Note: #49081 covers the same ground and was opened before this one. I only saw it after pushing, which is my fault for not checking first. Happy to close this in its favour if you prefer that one, entirely your call.

vasqu · 2026-09-24T12:56:18Z

+        # The Gemma 1.0 checkpoints were released with `hidden_act="gelu"`, which resolves to the exact
+        # erf GELU, but they were trained with the tanh approximation. `GemmaMLP` used to correct this
+        # by reading `hidden_activation`; #35235 dropped that field and left the legacy value in force.
+        # Remapping here rather than in the model means `save_pretrained` and anything reading the
+        # config (exporters, other runtimes) see the corrected value too.
+        if self.hidden_act == "gelu":
+            logger.warning_once(
+                '`hidden_act="gelu"` on a Gemma config is the legacy value from the original Gemma 1.0 '
+                "releases and resolves to the exact erf GELU, while these checkpoints were trained with "
+                'the tanh approximation. Setting `hidden_act="gelu_pytorch_tanh"` instead.'
+            )
+            self.hidden_act = "gelu_pytorch_tanh"


Suggested change

# The Gemma 1.0 checkpoints were released with `hidden_act="gelu"`, which resolves to the exact

# erf GELU, but they were trained with the tanh approximation. `GemmaMLP` used to correct this

# by reading `hidden_activation`; #35235 dropped that field and left the legacy value in force.

# Remapping here rather than in the model means `save_pretrained` and anything reading the

# config (exporters, other runtimes) see the corrected value too.

if self.hidden_act == "gelu":

logger.warning_once(

'`hidden_act="gelu"` on a Gemma config is the legacy value from the original Gemma 1.0 '

"releases and resolves to the exact erf GELU, while these checkpoints were trained with "

'the tanh approximation. Setting `hidden_act="gelu_pytorch_tanh"` instead.'

)

self.hidden_act = "gelu_pytorch_tanh"

# #35235 dropped this conversion which we now handle here instead

if self.hidden_act == "gelu":

logger.warning_once(

'We found `hidden_act="gelu"` in this Gemma config. This is a legacy value of the official '

'releases but it is meant to target the the tanh approximation. Setting `hidden_act="gelu_pytorch_tanh"` instead.'

)

self.hidden_act = "gelu_pytorch_tanh"

Let's keep it simple no?

vasqu · 2026-09-24T12:57:52Z



+@require_torch
+class GemmaLegacyActivationTest(unittest.TestCase):


no new class please, let's keep it simple like a mamba2 regression test were we check that loading with gelu leads to 1 the warning and 2 the config value being the tanh approx

@vasqu

Applies @vasqu's suggestion for the comment and the warning text, and replaces the separate test class with a single regression test in GemmaModelTest, following the diffusion_gemma CaptureLogger pattern: the warning fires, and the config value becomes the tanh approximation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

PCfVW · 2026-09-24T13:58:31Z

All three done, thanks.

I took your suggestion for the comment and the warning, with one doubled "the" removed. The test is now a single method in GemmaModelTest following the diffusion_gemma CaptureLogger pattern: the warning fires, and the config value becomes the tanh approximation.

vasqu · 2026-09-24T14:23:34Z

+
+        self.assertIn("gelu_pytorch_tanh", cl.out)
+        self.assertEqual(config.hidden_act, "gelu_pytorch_tanh")



Let's split the test into tester (like in mamba2)

Also we can capture the emitted warning message completely or a larger portion tbh

…ning Follows the mamba2 pattern: GemmaConfigTester(ConfigTester) with the check run from run_common_tests, wired in via setUp. The assertion is now on the complete emitted message rather than a fragment of it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

vasqu · 2026-09-24T15:04:32Z

huh the last commit doesnt make any sense why did we remove the workaround 😢

my comment on the tests is related to the test design like

transformers/tests/models/mamba2/test_modeling_mamba2.py

Lines 315 to 317 in 763a150

def test_mamba2_caching(self):

config_and_inputs = self.model_tester.prepare_config_and_inputs()

self.model_tester.create_and_check_mamba2_caching(*config_and_inputs)

Pls dont just aimlessly throw claude at it

PCfVW · 2026-09-24T15:07:53Z

Done. The test is now a GemmaConfigTester(ConfigTester) running from run_common_tests, as in mamba2, and it asserts the complete warning rather than a fragment.

Apologies for the first pass, I read "no new class" literally and put the method in GemmaModelTest.

vasqu · 2026-09-24T15:13:26Z

Please see https://github.com/huggingface/transformers/pull/49084/changes#r4095124979

…TY=error CI sets TRANSFORMERS_VERBOSITY=error (.circleci/create_circleci_config.py), so logger.warning_once emitted nothing and CaptureLogger captured an empty string. Wraps the capture in LoggingLevel(logging.WARNING), the same shape tests/generation/test_configuration_utils.py uses for its warning assertions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The __post_init__ remap was lost in 0042edc: a local mutation check had run `git checkout origin/main -- <source files>`, which updates the index as well as the working tree, and the follow-up commit staged only the test file. The source files were therefore committed back at their origin/main state while the working tree still held the fix, so every local run kept passing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

PCfVW · 2026-09-24T15:32:15Z

Sorry for the red. Two things, both mine.

The config change had been reverted out of the commit by a local mutation check that left my working tree intact, so I was testing something I had not actually pushed. Restored in 17e4cbe.

The test also needed LoggingLevel(logging.WARNING), since CI runs with TRANSFORMERS_VERBOSITY=error and the warning was being swallowed. Both are in now, and this time I verified from a clean checkout of the commit rather than from my working directory.

Moves the check onto GemmaModelTester as create_and_check_legacy_hidden_act_remap, with a short delegating test method on GemmaModelTest, matching the mamba2 shape at tests/models/mamba2/test_modeling_mamba2.py#L315-L317. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

PCfVW · 2026-09-24T16:26:48Z

Done, split the way you meant: create_and_check_legacy_hidden_act_remap on GemmaModelTester, with a short delegating method on GemmaModelTest, matching the mamba2 lines you linked. The warning assertion moved with it.

vasqu · 2026-09-24T17:00:36Z

run-slow: gemma

vasqu

Looks good sanity checking with the slow ci 🤗 thanks!

HuggingFaceDocBuilderDev · 2026-09-24T17:21:30Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

vasqu · 2026-09-24T17:37:41Z

Hmm I see that its those gemmas that are affected of the remap, I dont want to merge unless we find the commits that actually changed the output (after the regression) so we make sure that its actually as intended

PCfVW · 2026-09-25T08:50:20Z

Our 2 cents on this.

The expectations those tests use were re-recorded after the regression, so they encode the exact-GELU output.

#35235 landed as 2c47618c1a on 2024-12-18. The entry each failing test actually resolves to on the Nvidia runner was set later:

test	entry used on CUDA	set by
`test_model_2b_bf16`	`(None, None)`	`07848a8405`, 2025-05-26, #38384
`test_model_2b_4bit`	`("cuda", 8)`	`df0db63c2e`, 2026-08-14, #47976
`test_model_7b_fp16`	`("cuda", 8)`	`df0db63c2e`, 2026-08-14, #47976
`test_model_7b_bf16`	`("cuda", 8)`	`df0db63c2e`, 2026-08-14, #47976
`test_model_7b_4bit`	`("cuda", 8)`	`df0db63c2e`, 2026-08-14, #47976

Four of the five were recorded twenty months after the regression, specifically for A10G. Worth one caveat: test_model_2b_4bit still carries its original February 2024 text in its (None, None) fallback, but the ("cuda", 8) entry the runner picks is from August 2026.

More directly: those tests pass on main, which still has the bug, so the stored values are by definition what exact-erf-GELU Gemma emits rather than what the checkpoints were trained to emit.

The precision pattern points the same way. Only bf16, fp16 and 4bit moved; no F32 test changed. That matches the measurement in #49051, where the top-10 ranks are unchanged at F32 and 7 of 30 move at BF16.

I cannot regenerate the expected values myself, since they are per-device and I only have a consumer card.

vasqu · 2026-09-25T10:24:04Z

That's totally what I'm looking for! I was just wanting to make sure that the regression also forced updates on these slow tests and they did

I will update the values myself whenever I find time today 🫡

Talhax55z · 2026-09-25T10:39:01Z

Thanks for the heads-up @PCfVW, and congrats on getting this across the line — the review with @vasqu looks great.

Since #49084 is further along and already has maintainer engagement, I'll close this one in its favor. The fix matters more than whose commit it is.

Glad the bug is getting fixed either way. 🤗

github-actions · 2026-09-25T17:44:07Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: gemma

vasqu · 2026-09-25T17:45:18Z

run-slow: gemma

github-actions · 2026-09-25T17:46:44Z

Nvidia CI

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs on Nvidia:

models: ["models/gemma"]
quantizations: []

github-actions · 2026-09-25T17:47:14Z

AMD CI

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs on AMD:

models: ["models/gemma"]

github-actions · 2026-09-25T17:52:07Z

CI recap

Dashboard: View test results in Grafana
Latest run: 36168859151:2
Result: success | Jobs: 2 | Tests: 18 | Failures: 0 | Duration: 1m 35s

github-actions · 2026-09-25T18:19:11Z

CI Results (AMD)

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	ea874a1d	workflow commit (merge commit)
PR	2a81c19f	branch commit (from PR)
main	6b14ee96	base commit (on `main`)

Model CI Report

❌ 2 new failed tests from this PR 😭

gemma:
tests/models/gemma/test_modeling_gemma.py::GemmaIntegrationTest::test_model_7b_bf16 (❌ ⟹ ❌)
tests/models/gemma/test_modeling_gemma.py::GemmaIntegrationTest::test_model_7b_fp16 (✅ ⟹ ❌)

github-actions · 2026-09-25T19:01:39Z

CI Results (Nvidia)

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	ea874a1d	workflow commit (merge commit)
PR	2a81c19f	branch commit (from PR)
main	6b14ee96	base commit (on `main`)

✅ No failing test specific to this PR 🎉 👏 !

…inistic_algorithms Static cache triggers torch.compile which uses the Triton autotuner. The autotuner benchmarks multiple kernel configs based on GPU timing, which is non-deterministic and can flip a logit near a decision boundary introduced by PR #49084 (gelu_pytorch_tanh). torch.use_deterministic_algorithms(True) forces a single deterministic kernel path, making the output stable across runs and machines (verified: 6/6 stable on flaky A10G). Also removes the redundant ("cuda", 8) entry and the output truncation workaround. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…p16_static_cache Static cache triggers torch.compile which uses the Triton autotuner. The autotuner benchmarks multiple kernel configs based on GPU timing, which is non-deterministic and can flip a logit near the decision boundary introduced by PR #49084 (gelu_pytorch_tanh). Replacing the XPU-only decorator with require_deterministic_for_accelerator ensures deterministic kernel selection on all accelerators (CUDA, XPU, etc.), making the output stable across runs and machines (verified: 6/6 stable on flaky A10G runner). Also removes the redundant ("cuda", 8) Expectations entry and the output-truncation workaround that was masking the flakiness. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

… after huggingface#49084 (huggingface#49133) * [Gemma] Fix test_model_7b_fp16_static_cache expected value for cuda 8 after activation fix PR huggingface#49084 corrected the Gemma 1 hidden_act from "gelu" to "gelu_pytorch_tanh" (the activation the model was actually trained with). This changes the generated output on A10G (cuda, 8): "1999 4.0L 4x4" → "1995 4.0L 4x4" for test_model_7b_fp16_static_cache. Verified 30/30 runs on an A10G runner. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * revert: restore 1999 expected value, add dummy comment to retrigger CI Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [Gemma] Fix test_model_7b_fp16_static_cache flakiness with use_deterministic_algorithms Static cache triggers torch.compile which uses the Triton autotuner. The autotuner benchmarks multiple kernel configs based on GPU timing, which is non-deterministic and can flip a logit near a decision boundary introduced by PR huggingface#49084 (gelu_pytorch_tanh). torch.use_deterministic_algorithms(True) forces a single deterministic kernel path, making the output stable across runs and machines (verified: 6/6 stable on flaky A10G). Also removes the redundant ("cuda", 8) entry and the output truncation workaround. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [Gemma] Use require_deterministic_for_accelerator for test_model_7b_fp16_static_cache Static cache triggers torch.compile which uses the Triton autotuner. The autotuner benchmarks multiple kernel configs based on GPU timing, which is non-deterministic and can flip a logit near the decision boundary introduced by PR huggingface#49084 (gelu_pytorch_tanh). Replacing the XPU-only decorator with require_deterministic_for_accelerator ensures deterministic kernel selection on all accelerators (CUDA, XPU, etc.), making the output stable across runs and machines (verified: 6/6 stable on flaky A10G runner). Also removes the redundant ("cuda", 8) Expectations entry and the output-truncation workaround that was masking the flakiness. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

vasqu reviewed Sep 24, 2026

View reviewed changes

PCfVW and others added 2 commits September 24, 2026 17:21

nits

97540ed

vasqu approved these changes Sep 24, 2026

View reviewed changes

Talhax55z mentioned this pull request Sep 25, 2026

Fix Gemma 1.0 legacy GELU activation mapping in config #49081

Closed

fix

66a2535

nit

2a81c19

vasqu added this pull request to the merge queue Sep 25, 2026

Merged via the queue into huggingface:main with commit 27166ea Sep 25, 2026
40 of 41 checks passed

ydshieh mentioned this pull request Sep 27, 2026

[Gemma] Fix test_model_7b_fp16_static_cache expected value for cuda 8 after #49084 #49133

Merged

stevhliu mentioned this pull request Oct 23, 2026

[docs] Translation workflow #49889

Closed



		@require_torch
		class GemmaLegacyActivationTest(unittest.TestCase):


		self.assertIn("gelu_pytorch_tanh", cl.out)
		self.assertEqual(config.hidden_act, "gelu_pytorch_tanh")

	def test_mamba2_caching(self):
	config_and_inputs = self.model_tester.prepare_config_and_inputs()
	self.model_tester.create_and_check_mamba2_caching(*config_and_inputs)

Conversation

PCfVW commented Sep 24, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PCfVW commented Sep 24, 2026

Uh oh!

vasqu Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

vasqu Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

vasqu Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

PCfVW commented Sep 24, 2026

Uh oh!

vasqu Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

vasqu Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

vasqu Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

PCfVW commented Sep 24, 2026

Uh oh!

vasqu commented Sep 24, 2026

Uh oh!

PCfVW commented Sep 24, 2026

Uh oh!

PCfVW commented Sep 24, 2026

Uh oh!

vasqu commented Sep 24, 2026

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

HuggingFaceDocBuilderDev commented Sep 24, 2026

Uh oh!

vasqu commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PCfVW commented Sep 25, 2026

Uh oh!

vasqu commented Sep 25, 2026

Uh oh!

Talhax55z commented Sep 25, 2026

Uh oh!

github-actions Bot commented Sep 25, 2026

Uh oh!

vasqu commented Sep 25, 2026

Uh oh!

github-actions Bot commented Sep 25, 2026

Nvidia CI

Uh oh!

github-actions Bot commented Sep 25, 2026

AMD CI

Uh oh!

github-actions Bot commented Sep 25, 2026

CI recap

Uh oh!

github-actions Bot commented Sep 25, 2026

CI Results (AMD)

Commit Info

Model CI Report

Uh oh!

github-actions Bot commented Sep 25, 2026

CI Results (Nvidia)

Commit Info

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

PCfVW commented Sep 24, 2026 •

edited by github-actions Bot

Loading

vasqu commented Sep 24, 2026 •

edited

Loading