Repository navigation
StableAdamW: Missing divisor in RMS computation #27
Description
Hello
It seems as though there is a minor error in computing RMS in Stable Adam:
RMS is computed as:
And is missing a normalization coefficient for the mean:
rms = torch.norm(p.grad.data.div(root_sqr_avg.maximum(eps_t)), 2) / math.sqrt(p.grad.data.div.numel())
as per
Additionally, another eps can be added in the final normalization.
Activity
warner-benjamin commentedon Jun 3, 2024
I'm in the process of porting over the fastxtend optimizers to use the optimi implementations, which should resolve this.
warner-benjamin commentedon Jun 9, 2024
@EricZimmermann #28 tracks the progress of porting over the fastxtend optimizers to use the optimi implementations