NerdyRodent
patreon
Regional LoRAs in ComfyUI
🕑 Added 2024-12-14 13:56:10 +0000 UTCVideo - https://youtu.be/dASMuSCCNSU
Ever wanted to use one LoRA for part of your image, and another totally different LoRA (or model) for another?
Well, if you've seen the recent ComfyUI blog post about some new nodes for adding LoRAs to conditioning areas, along with various other nodes too, then you'll know this is now a lot easier to do!
The blog quite is quite long and goes into some depth, so instead I figured I'd start off a lot more basic and kinda work up from there ;)
Example workflow combining Redux & using a style LoRA on just one part:
Example output (workflow 8):
Links
Workflows
Loads of workflows this time! They are therefore numbered this time, starting off with the most basic and working up to the more involved ones.
Workflow 1 is a very basic version of regional prompting using the new nodes, with workflow 8 mixing both Fluxtapoz regional conditioning via the model, along with a second set of regional conditioning using a LoRA, like the example above.
Do note that the more conditioning you have, the more it's going to have to compute - like when you add a second conditioning to Flux in the form of a negative prompt.
Adding a LoRA to an area also then increases the time required, though this can be reduced slightly through use of timesteps.
Comments
Nerdy Rodent
Yes, also a good option!
Hussein
thanks but in this case ill create new Cond Set Props and Cond Set Combine to link 2 clips from 2 LoRAs to it? cuz i see Cond Set Props take only one Clip Text input
Nerdy Rodent
4_Flux-Comfy-Regional-Basic-LoRA_v0.json is the simple LoRA example. All you need to do is add both the clip and the LoRA hook to each character prompt
Hussein
Hi thanks for your informative stuff, but what if i have background and and have 2 LoARs characters models and i want compose them with the background using mask and region flux workflow, which workflow from these can help in that??
Swettkittens123
Hi NerdyRodent. Thank you so much for these workflows and the tutorial. Fantastic work! I would like in the flux Lora workflow to apply a controlnet depth, to the overall image. However in Comfyui the Apply Flux Controlnet node has as an output “controlnet_condition”, and not “Conditioning”, hence I can’t find a way to make it work with the Sampler node group in your example. Any chance you could guide me?
Nerdy Rodent
Looks like the default node has now been updated :)
blynx
Thank you! It really works, but it takes a long time. I opened an issue (#6295) to address this or to learn how to improve the efficiency. They have already seen it (#6296).
Nerdy Rodent
I'd be tempted to say that's a bug with the default node, as it's the only node causing issues. It probably didn't get updated like the rest. For now, any default node can just be replaced with the usual combine with mask. I've added a v1 for example 4 where the default node has been replaced with the standard masked combine.
blynx
For me, this structure is no longer working. The sigma update directly affects the 'model options'. Error message: "ComfyUI\comfy\samplers.py", line 227, in _calc_cond_batch finalize_default_conds(model, hooked_to_run, default_conds, x_in, timestep) TypeError: finalize_default_conds() missing 1 required positional argument: 'model_options'." I fixed it by adding it to the hook kf, but I’m not sure if this is the best solution, especially in cases where I only use one sampler. It also results in many more nodes. :(
Nerdy Rodent
The numbers show the order they appear in the video, starting off with a simple example. As the examples progress, you can see how easy it is to just add a lora to any existing workflow you may have, as well as combining it with areas from the "model" rather than "conditioning".
Nerdy Rodent
What is it that you think needs updating?
blynx
hi. are you going to update the workflows after yesterday's update? Fix Hook Keyframe 'guarantee_steps' behavior and add 'sigmas' (#6273).
Danny
What are the difference to the workflows a few videos ago? I am getting so confused by this many workflows :D
Asyouare
And also thank you for NOT saying: Today's sponsor....manscape. Oh gooooooooood.
Nerdy Rodent
:) And thank you!
Asyouare
Dear Nerdy, Thank you for the past year. The effort you have put into this for the benefit and enjoyment of all of us. I hope everything works out well for you as the year winds down and throught the next. But after that....kidding.
Nerdy Rodent
4 now has an SDXL example, as requested!
Clayfacer
That would be great
Nerdy Rodent
Yup! Or three, or four, or...
Samuel
Can we got two different subjects in the same image with regional LORA?
Nerdy Rodent
Yup! You can just swap the model loader to whatever - SD1.5, Pixart, SDXL, etc :) If you need me to upload an SDXL example, I can do that
Nerdy Rodent
Yup, unfortunately :/
EruoAuri
Hi NerdyRodent, do we still need to disable xformers for the flutapoz to work? Thanks
Clayfacer
Does this work for sd15 and sdxl? Any plans for workflows for those older models?
