enable test_layerwise_casting_memory cases on XPU#11406

a-r-r-o-w merged 5 commits into

huggingface:mainhuggingface/diffusers:mainfrom

yao-matrix:layerwise-xpuyao-matrix/diffusers:layerwise-xpuCopy head branch name to clipboard