Add additional logic to avoid load being called on every advance

[andrewcoh]

Proposed change(s)

#4931

I think i can still clean this up but im running on the cloud to see if we get a speedup/ensure correctness.

Useful links (Github issues, JIRA tickets, ML-Agents forum threads etc.)

Types of change(s)

• Bug fix
• New feature
• Code refactor
• Breaking change
• Documentation update
• Other (please describe)

Checklist

• Added tests that prove my fix is effective or that my feature works
• Updated the changelog (if applicable)
• Updated the documentation (if applicable)
• Updated the migration guide (if applicable)

Other comments

[dongruoping]

Good if the cloud experiments results also said so