You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am observing different outputs on the same input (only between the first run and the second one, the subsequent ones agree with the second oe). The following code reproduces the problem.
Slight numerical imprecision is sometimes inevitable, especially when dealing with half-precision models. A workaround is to "warm up" the model as you did, and after a third call or so it'll more likely emit deterministic outputs.
Thanks for more clues; I observed that too. I don't fully grasp what is happening under the hood, but will keep this issue open in case there's a fix possible, potentially in a future PyTorch version.
By the way, the problem is much worse under pytorch 1.7.0 (the maximum absolute difference goes up to 2.7236), and quite interestingly, when I use it for zero-shot classification only the first output (image_features_1) translates to sensible results. I suggest putting an assert on the version number.
Activity
jongwook commentedon Jan 13, 2021
Hi, thanks for reporting this issue.
I suspect it's due to CUDA's nondeterministic behavior, as it doesn't happen in the CPU mode.
Slight numerical imprecision is sometimes inevitable, especially when dealing with half-precision models. A workaround is to "warm up" the model as you did, and after a third call or so it'll more likely emit deterministic outputs.
josipd commentedon Jan 13, 2021
Thanks for the fast reply! I observe the same behavior even with
torch.set_deterministic(True)at the top andCUBLAS_WORKSPACE_CONFIG=:4096:8.jongwook commentedon Jan 13, 2021
Thanks for more clues; I observed that too. I don't fully grasp what is happening under the hood, but will keep this issue open in case there's a fix possible, potentially in a future PyTorch version.
josipd commentedon Jan 14, 2021
By the way, the problem is much worse under pytorch 1.7.0 (the maximum absolute difference goes up to 2.7236), and quite interestingly, when I use it for zero-shot classification only the first output (
image_features_1) translates to sensible results. I suggest putting an assert on the version number.jongwook commentedon Jan 14, 2021
Yeah that was why I needed to require 1.7.1. Putting an assert is a good idea, thanks!
josipd commentedon Jan 14, 2021
Btw, the above problem goes away on 1.7.0 with
jit=Trueinclip.load.dimitry12 commentedon Jan 22, 2021
My observation is that on pytorch 1.7.0 the after-warmup runs produce wrong results.
For example, the sample in README starts returning roughly equal score ~30% for each caption on second run and after:
pytorch 1.7.1 solves the problem
haltakov commentedon Feb 1, 2021
I've observed the same problem. Here two Colab notebooks if somebody wants to test:
With 1.7.0 the results on the second run are very far off.