Skip to content

Fix MaskedLayerNorm with bias=False - #4966

Open
adenzhou1350 wants to merge 1 commit into
pytorch:mainfrom
adenzhou1350:codex/fix-masked-layernorm-no-bias
Open

adenzhou1350 wants to merge 1 commit into
pytorch:mainfrom
adenzhou1350:codex/fix-masked-layernorm-no-bias

Conversation

@adenzhou1350

Copy link
Copy Markdown

MaskedLayerNorm inherits nn.LayerNorm's bias=False option, but its forward always adds self.bias. With affine weights enabled and bias disabled, that is None and forward raises a TypeError.

Apply the weight as before and only add bias when present. This leaves the existing treatment of zero-valued inputs and default affine behavior unchanged.

Tests cover bias-free forward/backward against nn.LayerNorm on nonzero inputs (float32/float64, one- and two-dimensional normalized shapes), mixed/all-zero inputs, and existing affine options.

Validation on Linux, Python 3.12 and PyTorch 2.11.0: normal TorchAO source-package import with C++ extensions disabled and CUDA hidden; python3 -m pytest test/prototype/pat/test_masked_layernorm.py -q passed (4 tests, 7 subtests). The same tests fail on main with the Tensor + None error. Ruff check/format and diff check pass. No GPU or full-repository CI claim.

Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
@pytorch-bot

pytorch-bot Bot commented Sep 30, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/4966

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 30, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant