The inference image is pure black. This seems to be related to weight, but I have been unable to pinpoint where the problem lies.

[ShanZard]

Describe the bug

I occasionally encounter situations where, when I restart my inference script again, the resulting image turns completely black. I haven't made any changes.

My intuition tells me that this is related to weight. I am currently a combination of SD and ControlNet, which I have trained on my own dataset. I can ensure that the ControlNet starts working, but sometimes it suddenly infers completely black images. I also checked some previous issues, but it doesn't seem to matter much. I sincerely hope you can guide me to locate the specific problem.

Reproduction

from PIL import Image
import torch
import os
import json
import re
from tqdm import tqdm
import numpy as np

negative_prompt = "blurry, not remote sensing image, distorted, GSD incorrect"
controlnet_model_name_or_path = '/opt/data/workspace/shanzhe/aaaitest/tempsave/controlnet-model-rule/checkpoint-31000'
pretrained_model_name_or_path = '/opt/data/workspace/shanzhe/shanzhemodels/stable-diffusion-3-medium-diffusers'
layout_path = '/opt/data/workspace/shanzhe/igmtestdataset/loveda_600/converted_masks/2_0_512_512_1024.png'

torch_dtype = torch.float16

controlnet = SD3ControlNetModel.from_pretrained(
controlnet_model_name_or_path,
subfolder="controlnet",
torch_dtype=torch_dtype,
)
pipe = StableDiffusion3ControlNetPipeline.from_pretrained(
pretrained_model_name_or_path,
controlnet=controlnet,
torch_dtype=torch_dtype,
).to("cuda")

layout_image = Image.open(layout_path).convert("RGB")
prompt = 'This ultra-high precision remote sensing image, provides a detailed view of a landscape characterized by a distinct juxtaposition of forested and agricultural areas.'
num_images = 2

generator = torch.Generator(device=pipe.device).manual_seed(2025)
images = pipe(
prompt=[prompt] * num_images,
negative_prompt=[negative_prompt] * num_images if negative_prompt else None,
control_image=[layout_image] * num_images,
generator=generator,
height=512,
width=512,
num_inference_steps=50,
guidance_scale=7.5,
max_sequence_length=512,
).images[0]
images.save("test_contrl_acc.png")

Logs

/opt/data/workspace/shanzhe/aaaitest/inference/checkblack.py 
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 10.06it/s]
Loading pipeline components...:  56%|████████████████████████████████████▋                             | 5/9 [00:01<00:00,  4.07it/s]You set `add_prefix_space`. The tokenizer needs to be converted from the slow tokenizers
Loading pipeline components...: 100%|██████████████████████████████████████████████████████████████████| 9/9 [00:01<00:00,  5.92it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:09<00:00,  5.17it/s]
/root/anaconda3/lib/python3.12/site-packages/diffusers/image_processor.py:148: RuntimeWarning: invalid value encountered in cast
  images = (images * 255).round().astype("uint8")

System Info

• 🤗 Diffusers version: 0.34.0.dev0
• Platform: Linux-6.11.0-26-generic-x86_64-with-glibc2.39
• Running on Google Colab?: No
• Python version: 3.12.7
• PyTorch version (GPU?): 2.7.1+cu126 (True)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Huggingface_hub version: 0.33.0
• Transformers version: 4.52.4
• Accelerate version: 1.7.0
• PEFT version: 0.15.2
• Bitsandbytes version: not installed
• Safetensors version: 0.5.3
• xFormers version: not installed
• Accelerator: NVIDIA L20, 46068 MiB
  NVIDIA L20, 46068 MiB
  NVIDIA L20, 46068 MiB
  NVIDIA L20, 46068 MiB
  NVIDIA L20, 46068 MiB
  NVIDIA L20, 46068 MiB
  NVIDIA L20, 46068 MiB
  NVIDIA L20, 46068 MiB
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help?

No response

[ShanZard]

In other words, I used the above script normally yesterday, but it was not normal today, which really confused me.

[ShanZard]

In addition, I also encountered this situation when using the official SD3 before, where I only needed to download the weights again, and they would work properly. Now I can't download the ControlNet weights again because they took me a lot of time to train.

[asomoza]

Hi, it would be really hard to help you here, there's a lot of unknowns for us, for example the controlnet which you don't provide, what type of controlnet it is and the mask.

Also the error is quite self explanatory, your model is producing invalid values, so you will need to check the tensor it produces and check what is wrong with it.

Also you mentioned that this happened to you with the official SD3? But before it was working? What you're describing as a fix implies that the download was corrupted, so it shouldn't have worked even once, if this isn't the case, you just got lucky that it "fixed" the problem because that's not a real fix unless the download was corrupted.

For your tests are you use the exact same parameters all the time, even the seed? So if you generate, let say 100 images with the same params and seed, some of them come out black, all of them, or it only happens if you change the seed?

In short, your black images can be caused by a lot of things, even bad hardware, so it's really hard to tell with what you're providing, if you want more help, you will need to upload the mask and the controlnet to the hub.

[ShanZard]

@asomoza Thank you for your reply. From the code, there are currently no issues. I kind of suspected it was the disk hardware, because it was so weird that the same weights and the same code would somehow break the next day, and I found a similar problem in the dataset, images can spoil over time

I can quickly detect the damage to the image, but I cannot see the damage to the weights because they can always be loaded normally. I hope you can guide me on how to check whether the weight is really normal.

Of course, I also uploaded a ControlNet weight that I trained, and I hope you can help me test it. I can make sure that this weight has been normal before, and I inferred a lot of normal images with the layout image and prompt.

https://huggingface.co/ShanZard/chechpointofcontrolnet (This is my trained ControlNet weight, and there is a mask in it, prompt: This ultra-high precision remote sensing image, provides a detailed view of a landscape characterized by a distinct juxtaposition of forested and agricultural areas.)
https://huggingface.co/stabilityai/stable-diffusion-3-medium-diffusers (The diffusion model that I used)

[asomoza]

Tested your model but I think your initial post is not accurate, with your controlnet, is not sometimes that it doesn't work, it doesn't matter what I do, it doesn't work, not even one time. I even did a loop, left it running for a long time and checked the latents.

Also it isn't the environment or library versions, I used other controlnets and they worked just fine.

Your controlnet returns NaNs on the control_block_samples which means that the model has some problem, if you inspect the checkpoint it seems ok at first glance, I can't do an exhaustive check since it requires a lot of time that I don't have, but at least it has all they keys and shapes which probably means that is not a corrupt model, that's why it loads.

I don't really know what you can do to fix, because as far as I know, there's nothing you can do. I suggest next time you do a backup, also you should save the model at different epochs because this can also happen with training (which would be the common cause for your case) and probably you should use a good SSD for this since it takes time to train a model.

[ShanZard]

@asomoza Thank you sincerely for your help! I have confirmed that it is a problem with my hard drive. Sorry for taking up a lot of your time. Fortunately, I found that I had backed up the checkpoint on another server, and everything is fine on this server.

However, my only doubt about this matter is which content in this checkpoint is damaged? Because it seems that all keys can be loaded in. If you have any suggestions on how to check this checkpoint content, please let me know. Thanks again for your help.