NerdyRodent
patreon
Flux.1 LoRA Training Tests - Llava Captioning
🕑 Added 2024-08-17 18:54:35 +0000 UTCFigured I'd try a different model, so this time it's llava!
Much the same idea as with using Florence to do the captioning, only this time I'm using LLM Party and Llava:
ollama pull llava-llama3
As with the Florence captioning you'll need to set 3 paths:
1. The directory with all your images
2. The full path to where you want to save the text files (e.g. /home/myuser/comfy/output/my-lora)
3. The path relative to your ComfyUI output directory to save the image files (e.g. my-lora)
Is it any better for training? Who knows! Give it a try and see... I know I am :)
Comments
Nerdy Rodent
I finally managed to re-create the issue :) ComfyUI recently changed up to CUDA 12.4 and I think it's llama_index using that old URL. llama-cpp-python doesn't look to be needed anyway, so it will run fine without it. They've also just made a few updates - https://github.com/heshengtao/comfyui_LLM_party/issues/82
Zenith Wu
Is this solved? I run into the same problem. Installing llama-cpp-python... Looking in indexes: https://jllllll.github.io/llama-cpp-python-cuBLAS-wheels/AVX2/cu124 ERROR: Could not find a version that satisfies the requirement llama-cpp-python (from versions: none) ERROR: No matching distribution found for llama-cpp-python
Guillaume C.
Yes. Very useful π
Nerdy Rodent
It's Comfyui_yanc - https://github.com/ALatentPlace/ComfyUI_yanc
Robert CODRON
Nerdy Rodent
Before it says "failed to import" the errors will be shown. Something like "can't do this", or "that thing failed" type messages. Another option is to make another comfy environment, install manager, then install llm party
SouthbayJay
I have checked the github but nothing there was helpful. No errors, just says βfailed to importβ Let me get back to the pc and see if there was anything else.
Nerdy Rodent
Have you checked the github issues? Also, any error messages at startup will provide information as to what's wrong with your environment
SouthbayJay
I wish I could use all these workflows with the LLM Part nodes but for the life of me I can't get them to install. It's a huge bummer..
Nerdy Rodent
May have to wait a while for whatever configuration you have, perhaps, if it's related to this one - https://github.com/abetlen/llama-cpp-python/issues/1682 ?
Nerdy Rodent
Not tried those, nope. Something to look at then - thanks! :)
Daniel Aleshinloye
Little off topics but have you tried Griptape nodes? They're a great set of custom nodes for working with different tools like LLMs, a little more streamlined for getting image captions IMO and linking multi prompt functions. And they support, OpenAI, Claude, Ollama, Florence2, and OpenAI endpoint style models. Thanks for the workflows, keep up the good work.
Arjan Speelman
I have trouble installing LLM Party, hope someone here happens to know the solution. Installing llama-cpp-python... Looking in indexes: https://jllllll.github.io/llama-cpp-python-cuBLAS-wheels/AVX2/cu124 ERROR: Could not find a version that satisfies the requirement llama-cpp-python (from versions: none) ERROR: No matching distribution found for llama-cpp-python The version seems to only go up to cu122.
Nerdy Rodent
I've been pretty impressed with llava captioning so far. 512 tokens, 1000 steps, 8e-5 and it's already got the details!
Andro.Meta
I've been playing with use kijai's Florence2 node and using cogflorence 2.1 - large with good results.
