[core] support attention backends for LTX

[sayakpaul]

What does this PR do?

Working code:

Unfold

import torch
from diffusers import LTXConditionPipeline
from diffusers.pipelines.ltx.pipeline_ltx_condition import LTXVideoCondition
from diffusers.utils import export_to_video, load_image, load_video

def round_to_nearest_resolution_acceptable_by_vae(height, width):
    height = height - (height % pipe.vae_spatial_compression_ratio)
    width = width - (width % pipe.vae_spatial_compression_ratio)
    return height, width

pipe = LTXConditionPipeline.from_pretrained(
    "Lightricks/LTX-Video-0.9.8-13B-distilled", torch_dtype=torch.bfloat16
).to("cuda")

image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/penguin.png")
video = load_video(export_to_video([image])) # compress the image using video compression as the model was trained on videos
condition1 = LTXVideoCondition(video=video, frame_index=0)
print(f"{image.size=}")

prompt = "A cute little penguin takes out a book and starts reading it"
negative_prompt = "worst quality, inconsistent motion, blurry, jittery, distorted"
expected_height, expected_width = 480, 832
downscale_factor = 2 / 3
num_frames = 96

# Part 1. Generate video at smaller resolution
downscaled_height, downscaled_width = int(expected_height * downscale_factor), int(expected_width * downscale_factor)
downscaled_height, downscaled_width = round_to_nearest_resolution_acceptable_by_vae(
    downscaled_height, downscaled_width
)
video = pipe(
    conditions=[condition1],
    prompt=prompt,
    negative_prompt=negative_prompt,
    width=expected_width,
    height=expected_height,
    num_frames=num_frames,
    num_inference_steps=8,
    generator=torch.Generator().manual_seed(0),
).frames[0]
export_to_video(video, "output.mp4", fps=24)

Output

output.mp4

Regarding the output:

1. I get the same output on main.
2. I didn't follow the three-stage pipeline of first generating low-res latents, upsampling them, and then running a few rounds of denoising. So, the quality loss is somewhat expected I believe (regardless of this PR).

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[a-r-r-o-w]

Thanks! Just some minor asks in the refactoring

[a-r-r-o-w]

super clean!

[sayakpaul]

Thanks @a-r-r-o-w! Have you started Wan already? If so, cool! If not, can I start and test your parallelism PR a bit with that?

[a-r-r-o-w]

Wan attention backend support has already been merged in #11918. For testing parallelism, I've tested most of the implementations (even some outside diffusers) to validate the soundness of going forth with CP-plans. Some plans are available here: https://github.com/huggingface/finetrainers/blob/f476c3717da6cbfb1070505a99ee989426b46d9c/finetrainers/models/_metadata/transformer.py#L68

[sayakpaul]

Cool! Once this gets matured a bit would love documenting some benchmakrs across the board.