NerdyRodent
patreon
Simple LLM Enhanced Flux!
🕑 Added 2024-10-12 13:07:48 +0000 UTCVideo Guide: https://youtu.be/qMNJ0dCoT_I
This is one for those who don't like noodles, tweaking things - or even moving around in the ComfyUI interface!
Everything you need for generating is all in once place, with shortcuts keys for the first-time use settings. This leaves you to focus (foooocus?) on prompting, with just the required options for:
Prompting
Image size
Seed
LLM prompting mode
HighRes mode
USD mode + resize factor
Example interface screenshot:
Press 1 = Model Settings
Press 2 = LLM Settings (optional)
The custom scheduler is a sort of mix between beta & simple which seems to gives the fewest stripes whilst also having reasonable detail. I've tested up to 2048x2048, so you can get fairly large outputs. Note that Ultimate Upscale takes a while!
Update Notes!
13/Oct/24 - The updated version (Flux-Simple-LLM-Ultimate_v2.json) has better switch logic, so now turning multiple things on or off at the same time should work properly :)
18/Oct/24 - Added Flux-Simple-LLM_v1.json, which is much the same as v0 but has a power lora loader and the LLM party node has been updated.
NB. The LLM party node was updated recently such that you'll need to:
Copy the existing system prompt text in the LLM Party node
right-click and select "Fix Node (recreate)" on any older workflows with this node.
Paste the system prompt text back in
The node has already been re-created in v1.
The same fix has been pre-applied to Flux-Simple-LLM-Ultimate_v2a.json
LLM Prompting Mode
The prompting mode is configured slightly differently here to enable "conversations". You can chat with Llama3.2 making prompts such as these possible:
Add more flowers
Give everyone a silly hat
What is to the left of them?
What is the main character thinking about?
Ollama
For the LLM Enhanced workflows you'll need to have ollama installed & running:
https://github.com/ollama/ollama
The workflow is configured to use llama3.2 by default, which you can download by running:
ollama pull llama3.2
Workflow Notes
Three different workflow versions are attached to this post:
Flux-Simple_v0.json = Simple interface, no LLM
Flux-Simple-LLM_v0.json = Simple interface + LLM
Flux-Simple-LLM-Ultimate_v0.json = Simple interface + LLM + Ultimate Upscale
If "Install Missing Custom Nodes" fails, they can be installed by using the search option instead:
Impact Pack
Extra Models for ComfyUI
ComfyUI-GGUF
ComfyUI_SUNoise
UltimateSDUpscale
Comfyroll Studio
rgthree's ComfyUI Nodes
ComfyUI Extra Samplers
Use Everywhere (UE Nodes)
KJNodes for ComfyUI
ComfyUI-ppm
pythongosssss/ComfyUI-Custom-Scripts
comfyui_LLM_party
Models
If you are missing any models, download them from here:
Flux = https://huggingface.co/black-forest-labs/FLUX.1-dev
Flux GGUF = https://huggingface.co/city96/FLUX.1-dev-gguf/tree/main
CLIP = https://huggingface.co/zer0int/CLIP-GmP-ViT-L-14
Hyper-SD = https://huggingface.co/ByteDance/Hyper-SD
Shakker Labs = https://huggingface.co/Shakker-Labs
Upscalers = https://openmodeldb.info/
Low VRAM
A GGUF loader (bypassed by default) is also available for those with low VRAM. Simply delete the standard loader then un-bypass and connect the GGUF one. The default configuration works best with 24GB+
Files to download for the regular version
From https://comfyanonymous.github.io/ComfyUI_examples/flux/ -
"If you donβt have t5xxl_fp16.safetensors or clip_l.safetensors already in your ComfyUI/models/clip/ directory you can find them on: this link. You can use t5xxl_fp8_e4m3fn.safetensors instead for lower memory usage but the fp16 one is recommended if you have more than 32GB ram.
The VAE can be found here and should go in your ComfyUI/models/vae/ folder."
Comments
droghtak
okey just for completion, I was using a FP8 model with vae and clips backed in... im so smart :D. Besides this, sometimes chrome hoards gazillions of VRAM for no reason. But everything is ok now :-)
SouthbayJay
Thank you
Nerdy Rodent
If you're using the legacy interface then you'll need to turn the new menus on via the settings cog.
Nerdy Rodent
You can either start over with a fresh environment, or debug your existing one. If there are any errors shown while comfy is starting up, that would be the place to start. Often a standard install can have fewer issues in the long run than the portable download, if you're using that. Anything outside the standard Linux + Nvidia setup can come with it's one, special issues too!
B. Elliott Baker
trying to use this but after i install Comfyroll Studio nodes my comfyui won't load the workflow... i just get a spinning wheel forever.
Loakamts
Itβs also possible to use openrouter as an llm supplier. While it does cost a bit, its pay as you go (and not expensive depending on the llm your using). I understand privacy might be a concern though. If youβre interested in the setup, let me know (itβs pretty quick and easy). π
droghtak
Damn it I have to recheck everything. I guess you are using regular comfiuy from official repo right? Perhaps I gave some troublesome node π€
Nerdy Rodent
Oddly enough, the Q8 GGUF actually uses more VRAM for me (as well as being slower XD). I've got two monitors attached, with my pre-comfy VRAM usage sitting at just under 400MB (225 standard, 300 if I open Chrome). Never had an OOM yet (even with larger LLMs)!
droghtak
MMM I'm using your ultimate V2 workflow on Ubuntu. Perhaps I'm doing something wrong but I'd say I'm using the same on everything. Are you using a headless system? Perhaps my X are eating too much VRAM
Nerdy Rodent
I'm on a 3090 here and it peaks up to 18GB VRAM, but with an initial image of 2048x2048 and HiRes enabled, it can go up to 19GB. Personally I avoid the GGUF models as they're so much slower.
droghtak
A 3090 is having hard time with this one, I need to load the gguf version which is a shame. I highly suggest to try the nf4 models, they work similar or even better than FP8
SouthbayJay
I have the new version but I'm missing a lot of the features.. Like the models folder on the left side bar, the default template menu and the floating queue button... I've updated a few times but those don't seem to appear nor do I see anything in the setting menu.
Nerdy Rodent
The option was added a few months ago now in settings -> use new menu, like I show here in a previous video - https://youtu.be/g8W3xe5kRBQ
SouthbayJay
Thank you for the workflows and the video! For some reason I'm not getting the new UI.. I have the bar on top and side but not the floating queue button or the models folder.. Anything special I need to do?
Nerdy Rodent
Ah, that's useful :/ I've cleared the links from there so that doesn't happen!
Nibmeister
Flux-Simple-LLM_v0 works really nicely on my 3070TI.
David Henrry Ticona
Hi Nerdy, trying to download the workflows, it says "Expired URL" for these links: Workflow Notes Three different workflow versions are attached to this post: Flux-Simple_v0.json = Simple interface, no LLM Flux-Simple-LLM_v0.json = Simple interface + LLM Flux-Simple-LLM-Ultimate_v0.json = = Simple interface + LLM + Ultimate Upscale can you please share them again?
Nerdy Rodent
The hires mode does indeed essentially use img2img for the upscale. Should modify a few details (hopefully for the better) along with the increased resolution!
Nerdy Rodent
The option was added a few months ago now in settings -> use new menu, like I show here - https://youtu.be/g8W3xe5kRBQ
timo
same
Rart
nice do we have img to img - mode with upscale ?
Clayfacer
Very cool.
Joe Blow
I've updated my ComfyUI but don't see an option for the new interface. Love the content BTW.
