[tests] tighten compilation tests for quantization#12002

sayakpaul wants to merge 6 commits intomainhuggingface/diffusers:mainfrom

tighten-compile-quant-testshuggingface/diffusers:tighten-compile-quant-testsCopy head branch name to clipboard