Repository navigation
chore: Validate eps in Adam at construction - #1
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adam and AdamW accept any eps, including negative values , while RMSprop, Adagrad, AdaDelta all reject eps <= 0 at construction. eps sits directly in the update denominator, so a bad value doesn't error, it just changes which direction the parameter moves
This adds one check to Adam.init, right after the existing betas validation. Since AdamW and Adamax both call this init through super(), the check covers all three optimizers at once, and Adamax's own copy of the check becomes dead code so i removed in this diff. The only visible side effect is that Adamax(eps=-1)'s error message now reads "Adam epsilon..." instead of "Epsilon value...".
The bound stays >=0 rather than >0, because Adamax already allows eps=0. Tightening that is a separate decision I'm leaving to maintainers, not something i should do in these fix. eps=0.0 and small positive values still construct for Adam, AdamW, and Adamax, unaffected by this change.
The new test cases fail on main with AssertionError: ValueError not raised, and pass on this branch under both DEVICE=cpu and DEVICE=gpu (26/26), with the full suite at 858 OK, 5 skipped, and pre-commit clean.
I used Claude Code to help me but every number and error string above is from a run on my machine, and I reviewed and understand every line in this diff.