-
Notifications
You must be signed in to change notification settings - Fork 2.6k
Integrate Bria 3.1/3.2 Models and ControlNet Pipelines into InvokeAI #8248
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
1076d8e
to
2d55dbe
Compare
Hey! Confirming, have you tested that this works with the new location? I ran into some issues and may need to poke around in my local install to see what's going on if you were able to run these without issue |
3af3004
to
c08a6a8
Compare
Hey! yeah had an import error, fixed it. Should work now |
37ebc28
to
29aed4b
Compare
Thanks - Trying to get this to run in between a lot of other stuff. I'm getting an error Device-side assertions were explicitly omitted for this error check; the error probably arose while initializing the DSA handlers. Looking at the above, it may be wise to reference some of the device references elsewhere -- Is the noise generation something that needs to happen on GPU? |
It would be significantly faster if it runs on the gpu, but I suspect that if it fails here it will also fail in the denoiser node. |
29aed4b
to
c296fd2
Compare
I am able to run the node now (progress) although it's extremely slow - I suspect that there are a number of missing hooks into existing device references, tools for memory management, etc. Did you reference other inference nodes for conventions, or is this primarily a simple code port? (Re: Device I'm on - testing on a box with 4 ADA 6000s) |
I referenced flux and sd3 inference nodes, but I didn't found anything special other then the use of |
a554237
to
8929143
Compare
@ilanbria - We're planning on merging this to begin work on the UI, however it's running into some testing failures now that our standard workflows are running. Let us know if any you need clarity on any of these |
2451c2c
to
711a579
Compare
Fixed the ruff and the schema, should work now |
711a579
to
8e43c31
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Massive effort on this, well done!
It does need some changes to fit in with Invoke's conventions and patterns.
- What is the latent scale factor for Bria 3.1/3.2? We will use this constant value in the Canvas to constraint the generation sizes.
- The ControlNet nodes are not compatible with Invoke's Canvas. Invoke will handle all "preprocessing" up-front and provide the control image directly to the ControlNet node. Canvas will resize the image to match the input noise/latents.
- Invoke has nodes to handle Canny edge detection, "color tile", depth detection, etc. We don't need to vendor in the code for that functionality.
- Also, as mentioned in a comment in the openpose code, Invoke is Apache-2.0 and not compatible with the explicitly non-commercial license in that openpose code.
- Does Bria support image-to-image workflows? If so, we'll need a VAE encoder node.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The logic in this file is the legacy implementation. We are not adding to it.
Please define the Bria model config in invokeai/backend/model_manager/config.py
. Write the classification logic in that config class, implemented with the matches()
and parse()
methods.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here's how to implement the Bria model probing logic in config.py
:
diff --git a/invokeai/backend/model_manager/config.py b/invokeai/backend/model_manager/config.py
index e90d64509e..7c35cea3d5 100644
--- a/invokeai/backend/model_manager/config.py
+++ b/invokeai/backend/model_manager/config.py
@@ -442,26 +442,6 @@ class LoRADiffusersConfig(LoRAConfigBase, ModelConfigBase):
"base": cls.base_model(mod),
}
-class BriaDiffusersConfig(LoRAConfigBase, ModelConfigBase):
- """Model config for Bria/Diffusers models."""
-
- format: Literal[ModelFormat.Diffusers] = ModelFormat.Diffusers
-
- @classmethod
- def matches(cls, mod: ModelOnDisk) -> bool:
- if mod.path.is_file():
- return cls.flux_lora_format(mod) == FluxLoRAFormat.Diffusers
-
- suffixes = ["bin", "safetensors"]
- weight_files = [mod.path / f"pytorch_lora_weights.{sfx}" for sfx in suffixes]
- return any(wf.exists() for wf in weight_files)
-
- @classmethod
- def parse(cls, mod: ModelOnDisk) -> dict[str, Any]:
- return {
- "base": cls.base_model(mod),
- }
-
class VAECheckpointConfig(CheckpointConfigBase, LegacyProbeMixin, ModelConfigBase):
"""Model config for standalone VAE models."""
@@ -541,6 +521,35 @@ class MainDiffusersConfig(DiffusersConfigBase, MainConfigBase, LegacyProbeMixin,
pass
+class BriaDiffusersConfig(DiffusersConfigBase, MainConfigBase):
+ """Model config for Bria/Diffusers models."""
+
+ format: Literal[ModelFormat.Diffusers] = ModelFormat.Diffusers
+ base: Literal[BaseModelType.Bria] = BaseModelType.Bria
+
+ @classmethod
+ def matches(cls, mod: ModelOnDisk) -> bool:
+ if mod.path.is_file():
+ return False
+
+ config_path = mod.path / "transformer" / "config.json"
+ if config_path.exists():
+ with open(config_path) as file:
+ transformer_conf = json.load(file)
+ if transformer_conf["_class_name"] == "BriaTransformer2DModel":
+ return True
+
+ return False
+
+ @classmethod
+ def parse(cls, mod: ModelOnDisk) -> dict[str, Any]:
+ return {}
+
+ @classmethod
+ def get_tag(cls) -> Tag:
+ return Tag(f"{ModelType.Main.value}.{ModelFormat.Diffusers.value}.{BaseModelType.Bria.value}")
+
+
class IPAdapterConfigBase(ABC, BaseModel):
type: Literal[ModelType.IPAdapter] = ModelType.IPAdapter
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For Invoke, "image-to-image" means partial denoising. We provide partially noised latents to the transformer or UNet and then only do some of the denoising steps. For example, our existing denoise nodes have fields for:
We use these parameters to add the appropriate amount of noise to the input latents and do partial denoising. ControlNet is handled separately, and can be used with text-to-image or image-to-image. |
Summary
This feature PR integrates Bria 3.1/3.2 text-to-image models and their ControlNet pipelines into InvokeAI.
How
Representative commits:
Add Bria text to image model and controlnet support
Setup Probe and UI to accept bria main/controlnet models
Added scikit-image required for Bria's OpenposeDetector model
Related Issues / Discussions
Follow-up to exploratory PR Setup Probe and UI to accept bria main models #7973 ("Setup Probe and UI to accept Bria main models").
QA Instructions
Steps:
Install Models
Set HF token with access to Bria's models.
Install
briaai/BRIA-3.2
.Install
briaai/BRIA-3.2-ControlNet-Union
.Load Workflow
Load the workflow from the attached file:
[Bria's ControlNet workflow.json](https://github.com/user-attachments/files/21248912/Bria.s.ControlNet.workflow.json) [Bria's text-to-image workflow.json](https://github.com/user-attachments/files/21248924/Bria.s.text-to-image.workflow.json)Run Workflow
Run workflow without controlnet.
Run workflow with controlnet.
Merge Plan
Rebase onto
main
after current release.Checklist
What's New
copy (if doing a release after this PR)