[1.1.454] ControlNet union SDXL model

[huchenlei]

ControlNet Union is a new type of ControlNet that can handle multiple input conditions

• https://huggingface.co/xinsir/controlnet-union-sdxl-1.0
• https://github.com/xinsir6/ControlNetPlus

How to use

Download model

Model can be downloaded here: https://huggingface.co/xinsir/controlnet-union-sdxl-1.0/tree/main. After download, it is recommended to rename the file as controlnet++_union_sdxl so that the control type filter can correctly handle it.

Openpose

Do not use key map with face and hand as the union controlnet does not seem to be trained with hand/face annotation

Canny

Depth

SoftEdge

Normal

Segmentation

Note: ofade20k preprocessor does not seem to work

Multi-ControlNet

Currently the multi-controlnet is not working in the optimal way described in the original paper, but you can still try use it, as it can help you save VRAM by avoid loading another controlnet for different type of control.

Unit1 setup

Unit 2 setup

Generation result

Appendix

Input images

Mask

Generation parameters (Parameters including seed are fixed for all examples)

1girl, sitting on throne,, absurdres, highres, (masterpiece:1.2), (best quality, highest quality),
Negative prompt: (lowres, low quality, worst quality:1.2), (text:1.2), watermark, (frame:1.2), deformed, ugly, deformed eyes, blur, out of focus, blurry, deformed cat, deformed, photo, anthropomorphic cat, monochrome, photo, pet collar, gun, weapon, blue, 3d, drones, drone, buildings in background, green
Steps: 30, Sampler: DPM++ 2M, Schedule type: Karras, CFG scale: 7, Seed: 1413879340, Size: 1024x1024, Model hash: 5d973c1f8a, Model: animagine-xl-2.0, ControlNet 0: "Module: none, Model: controlnet++_union_sdxl [15e6ad5d], Weight: 1.0, Resize Mode: Crop and Resize, Processor Res: 512, Threshold A: 0.5, Threshold B: 0.5, Guidance Start: 0.0, Guidance End: 1.0, Pixel Perfect: False, Control Mode: Balanced", ControlNet 1: "Module: canny, Model: controlnet++_union_sdxl [15e6ad5d], Weight: 1.0, Resize Mode: Crop and Resize, Processor Res: 512, Threshold A: 100.0, Threshold B: 200.0, Guidance Start: 0.0, Guidance End: 1.0, Pixel Perfect: False, Control Mode: Balanced", Version: v1.9.4-249-g93c00b2a

[sdbds]

xinsir6/ControlNetPlus#1
According to my issue, union used mobile sam random split instead of ADE20K

Considering the controlnet_aux repository is now hosted by huggingface, and more new research papers will use the controlnet_aux package, I think we can talk to @Fannovel16 about unifying the preprocessor parts of the three projects to update controlnet_aux.

[Fannovel16]

@sdbds It's quite challenging as comfyui_controlnet_aux has a different DWPose implementation and embeds many third party packages's fork

[sdbds]

Yes, I'm going to start experimenting with preprocessors that don't rely on third-party packages, and I'll just use what's in the repository for the rest of it.
Also controlnet_aux's DWpose implementation seems to be faster and more suitable for video frame conversion, in my previous tests.

[slashedstar]

CAn we have a update for the promax version with tile and inpainting? 👀

[huchenlei]

Tile supported, but inpaint still has issue. #2998

Need to wait official inference script for inpaint from union controlnet author to see what is wrong for inpaint.

[TKoestlerx]

Edit: Outpaint is working with the preprocessor set to "none".
Inpaint is not working correctly as it seems to draw information from the masked out area.

Edit2: You can do inpainting if you use the "Effective Region Mask" Feature to upload the mask instead of the default inpainting map.

[Gavr728]

I tried to do outpainting with the preprocessor set to none, but surprisingly it also changed colors on the original picture while preserving all the other details in tact.

[sdbds]

Tile supported, but inpaint still has issue. #2998

Need to wait official inference script for inpaint from union controlnet author to see what is wrong for inpaint.

They put on other place...
https://github.com/xinsir6/ControlNetPlus/blob/main/promax/controlnet_union_test_inpainting.py

[AviceProjects]

Any updates to fixing inpainting? Maybe with what sdbds mentioned?

[ahmed20247]

Does it work with Domfyui?