sebastiankamph
patreon
Flux ControlNet - Ultimate Workflow
🕑 Added 2024-10-17 15:15:30 +0000 UTCVideo guide https://youtu.be/B3HBAiGuySk
In this guide we’ll go over a workflow for using Flux ControlNet within Comfy. You will be able to easily use one or more ControlNets just by flipping a switch. ControlNet will control your output image by the input image you load.
If you don't have Comfy or the Comfy Manager installed, you can find a guide on it here: https://www.youtube.com/watch?v=KTPLOqAMR0s
Drag and drop the .json file (download at bottom) into your Comfy. If you get red nodes, make sure to go into your Manager and Install all missing custom nodes.
Install Flux ControlNet models by clicking Model Manager and then search for ControlNet Union Pro. Use the larger one if you don't have vram or storage issues. The smaller fp8 one can be used for lower end machines. These models go in the /models/ControlNet/ folder
First, load your preferred image in the Load image node and select the size you want your output image to be. Your input image will be resized accordingly.
Select your preferred model GGUF model. By default in this workflow we're using Flux Dev Q8 GGUF, which is the quant with highest quality. If you don't have GGUF models, you can find them in the Model Manager (as seen above for ControlNets, search for GGUF). These models go in the /models/unet/ folder
The VAE and CLIP can also be found through the model manager.
Also available at this link: https://huggingface.co/comfyanonymous/flux_text_encoders/tree/main
ae.safetensors go in /models/vae/
tx5xxl_fp16 and clip_l go in /models/clip/
If you want to load a Lora, you can select these from the Load Lora nodes. To disable or enable loras, right click a node and select “Bypass node” or select it and press Ctrl + B / Cmd + B. When they are purple, they are disabled.
Type your prompt in the positive (green) node. This is where you type what you want to see in your image. For this Flux workflow, negative prompting is not enabled.
These nodes are your ControlNets and they are split up into several groups. First, a ControlNet model is loaded. We’re also loading a preprocessor to look at the input image and send that image preprocessed together with the ControlNet model into the Apply ControlNet node. This is where you would finetune your settings if needed.
Recommended values for Flux dev ControlNet Union Pro:
Strength around 0.5-0.7
End_percent around 0.1-0.2
Strength: How much impact this ControlNet will have on your result, set 0-1
Start_percent: When in the generation ControlNet will activate, set 0-1
End_percent: When in the generation ControlNet will finish, set 0-1
Example A: If your generation is running for 20 steps (Ksampler value) and you set end_percent to 0.5, then ControlNet will be active for 50% of that generation time, 10 steps. The last 10 steps will run without ControlNet.
Example B: Your generation is set at 20 steps in the Ksampler. Start_percent is set at 0.5 and end_percent at 0.8. Your generation will run 10 steps without ControlNet. It will activate after 10 steps and run with ControlNet and then disable again after 16 steps to finish the last 4 steps without ControlNet.
To enable or disable a ControlNet group, click the “Fast Bypasser” node in the right corner which says Enable yes/no
Your output will be in this group together with the Ksampler. The Ksampler is where the actual generation or rendering takes place.
Set the amount of steps you want.
Around 10-20 for euler
CFG is based on the model you use. We're using Flux so set CFG at 1
SD 1.5 models use between 3-7, but specialty models like Flux, SD-LCM/Lightning/Turbo etc. will use a lower CFG like 1-2. Check your model information.
Denoise is how much of your input image will change. This is not based on ControlNet, but more similar to img2img. If you lower the denoise value, you will retain parts of your latent_image input. However in this workflow our latent_image input is just an empty node so nothing will be retained. Keep value at 1.
Seed is set at randomize, and gives you a random image each time you click generate. If you want to finetune your settings, you can set this to fixed to see the changes you make for each generation.
As an added bonus, there is an ultimate SD Upscale node to upscale your results. Recommended is to leave this disabled at first, as the extra generation time will be quite resource intensive.
Feel free to play with the values to adjust the upscaling, try different upscaling models, steps and samplers to find the settings for your image.
It currently is upscaling in tiles. Recommended value for tiles in Flux are 1024x1024
You can download the NOMOS Upscaler model at https://openmodeldb.info/models/4x-Nomos8kDAT
Place in your /models/ESRGAN/ folder
TROUBLESHOOTING
Model selection nodes turn red
Make sure your models are in the right folder. Refresh ComfyUI and re-select the model in the dropdown list of the red border node.
Image is weird or ugly
Turn down ControlNet strength or end_percent
Workflow crashes or goes out of memory
Load fp8 ControlNet Union Pro.
Load fp8 text encoders.
Load a smaller Flux dev GGUF model (like Q1-5)
Load fewer ControlNets (disable on the switch)
Comments
Sergio Orellana
Hi, im having this error: "Error occurred when executing ControlNetLoader / MMDiT.__init__() got an unexpected keyword argument 'image_model'"
FRaone
Anyone else with Numpy 2 compatibility issues?} Compatibility issues with newbyteorder #5451 https://github.com/comfyanonymous/ComfyUI/issues/5451
Sebastian Kamph
Uploading a fix to this. Some versions of Comfy is not compatible with the individual switch.
Travis Little
This was a great video and well-structured workflow, but the fast switch does not work on my system. It does not "un-bypass" the nodes in the group -- just the reroute.
Nibmeister
Works really well. Thanks!
Delambo Culus
Thank you!!! Please do another when Control Net comes to WebForge for Flux?
