Post "SOTA Image Captioning Scripts For Stable Diffusion: CogVLM, LLaVA, BLIP-2, Clip-Interrogator (115 Clip Vision Models + 5 Caption Models)" by "Furkan Gözükara" from patreon

Furkan Gözükara

no error it is running

Hockey

I get errors when installing Llava v5. ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. llava 1.2.2.post1 requires gradio==4.16.0, but you have gradio 4.44.1 which is incompatible. llava 1.2.2.post1 requires gradio_client==0.8.1, but you have gradio-client 1.3.0 which is incompatible. llava 1.2.2.post1 requires httpx==0.24.0, but you have httpx 0.28.1 which is incompatible. llava 1.2.2.post1 requires torch==2.1.2, but you have torch 2.2.0+cu121 which is incompatible. llava 1.2.2.post1 requires torchvision==0.16.2, but you have torchvision 0.17.0+cu121 which is incompatible. and then Windows Run Part 1: LINK : fatal error LNK1181: cannot open input file 'aio.lib' test.c LINK : fatal error LNK1181: cannot open input file 'cufile.lib' [2025-04-17 11:11:34,174] torch.distributed.elastic.multiprocessing.redirects: [WARNING] NOTE: Redirects are currently not supported in Windows or MacOs. 2025-04-17 11:11:34 | INFO | controller | args: Namespace(host='127.0.0.1', port=10000, dispatch_method='shortest_queue') 2025-04-17 11:11:34 | INFO | controller | Init controller 2025-04-17 11:11:34 | ERROR | stderr | INFO: Started server process [13260] 2025-04-17 11:11:34 | ERROR | stderr | INFO: Waiting for application startup. 2025-04-17 11:11:34 | ERROR | stderr | INFO: Application startup complete. 2025-04-17 11:11:34 | ERROR | stderr | INFO: Uvicorn running on http://127.0.0.1:10000 (Press CTRL+C to quit)

Furkan Gözükara

amazing

Ec Jep

it works extremely well. It has been very helpful with accurately describing scenes and helps me write prompts better since I can follow how it prioritizes people, place, objects and scene descriptions

Furkan Gözükara

i think joy is best currently

Ec Jep

which captioning do you recommend? I recently used your post to install Joy and it works well. How are these different/better?

Furkan Gözükara

you are in inaccurate folder: /home/Ubuntu/Desktop/Cog/web_app_CogVLM.py' you have to be in after activating venv /home/Ubuntu/Desktop/Cog/CogVLM/web_app_CogVLM.py watch video and you will do it

mike oxmaul

files are in /home/Ubuntu/desktop/cog/ CogVLM folder exists after runing the installer without error Your installer and start scripts are located in home/Ubuntu/desktop/cog/

mike oxmaul

sorry. cannot see what i'm doing wrong here. Ubuntu@0009-kci-prxmx10100:~/Desktop/Cog$ cd CogVLM source venv/bin/activate cd .. chmod +x Mass_Compute_Start.sh ./Mass_Compute_Start.sh Please select an option: 1. Start As 4-bit Precision 2. Start As 8-bit Precision 3. Start As 16-bit Precision 4. Start As 32-bit Precision Enter your choice (1-4): 1 Please select an option: 1. Start Without Gradio Share 2. Start With Gradio Share Enter your choice (1-2): 1 python3: can't open file '/home/Ubuntu/Desktop/Cog/web_app_CogVLM.py': [Errno 2] No such file or directory Press enter to continue... (venv) Ubuntu@0009-kci-prxmx10100:~/Desktop/Cog$

Furkan Gözükara

please watch between 6:20 and 9:20 of this video to learn how to install and run massed compute scripts : https://youtu.be/wG7oPp01COg

mike oxmaul

(venv) Ubuntu@0009-kci-prxmx10100:~/Desktop/Cog$ chmod +x Mass_Compute_Start.sh (venv) Ubuntu@0009-kci-prxmx10100:~/Desktop/Cog$ ./Mass_Compute_Start.sh Please select an option: 1. Start As 4-bit Precision 2. Start As 8-bit Precision 3. Start As 16-bit Precision 4. Start As 32-bit Precision Enter your choice (1-4): 2 Please select an option: 1. Start Without Gradio Share 2. Start With Gradio Share Enter your choice (1-2): 1 python3: can't open file '/home/Ubuntu/Desktop/Cog/web_app_CogVLM.py': [Errno 2] No such file or directory

mike oxmaul

thanks mate. trying now

Furkan Gözükara

Yes follow the Massed_Compute_Instructions_READ.txt. i just tested on massed compute and works great

mike oxmaul

fyi - you've got mass_compute_start and massed_compuete_install mass vs massed

Furkan Gözükara

with V9 massed compute added. I am also testing right now. will update again if not works

Furkan Gözükara

ok testing and fixing now

mike oxmaul

its from your cogvlm_v8

Furkan Gözükara

which script? i will update and fix it. massed compute uses python3 as command not python

mike oxmaul

Currently using your Massed Compute VM

Furkan Gözükara

That means you dont have python in your system. You have to install Python 3.10, CUDA 11.8, Nvidia drivers and C++ tools manually. Where do you use? Your local Ubuntu computer?

mike oxmaul

install_Linux.sh: 9: python: not found Composing venv install_Linux.sh: 13: python: not found install_Linux.sh: 19: source: not found When running sh install_Linux.sh

Furkan Gözükara

Ye that is a great gpu same as what I use. Follow video exactly same and reinstall and it should work

Walker4k

I'll have a look at the YT vid. GPU is NVIDIA GeForce RTX 3090

Furkan Gözükara

ok i did a fresh install and works for me. that means you have python cuda install error. here the video that you need to follow exactly : https://youtu.be/-NjNy7afOQ0 and what is your GPU model?

Furkan Gözükara

installing right now to test

Walker4k

Following error when attempting to use clip interrogator (ViT-L-14/openai + blip-large) - Traceback (most recent call last): File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\transformers\feature_extraction_utils.py", line 182, in convert_to_tensors tensor = as_tensor(value) File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\transformers\feature_extraction_utils.py", line 141, in as_tensor return torch.tensor(value) RuntimeError: Could not infer dtype of numpy.float32 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\gradio\queueing.py", line 541, in process_events response = await route_utils.call_process_api( File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\gradio\route_utils.py", line 276, in call_process_api output = await app.get_blocks().process_api( File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\gradio\blocks.py", line 1928, in process_api result = await self.call_function( File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\gradio\blocks.py", line 1514, in call_function prediction = await anyio.to_thread.run_sync( File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\anyio\to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 2177, in run_sync_in_worker_thread return await future File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 859, in run result = context.run(func, *args) File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\gradio\utils.py", line 833, in wrapper response = f(*args, **kwargs) File "O:\captioning\captioners_clip_interrogator_v2\Clip_Interrogator.py", line 118, in image_to_prompt return ci.interrogate(image) File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\clip_interrogator\clip_interrogator.py", line 244, in interrogate caption = caption or self.generate_caption(image) File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\clip_interrogator\clip_interrogator.py", line 191, in generate_caption inputs = self.caption_processor(images=pil_image, return_tensors="pt").to(self.device) File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\transformers\models\blip\processing_blip.py", line 103, in __call__ encoding_image_processor = self.image_processor(images, return_tensors=return_tensors) File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\transformers\image_processing_utils.py", line 551, in __call__ return self.preprocess(images, **kwargs) File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\transformers\models\blip\image_processing_blip.py", line 310, in preprocess encoded_outputs = BatchFeature(data={"pixel_values": images}, tensor_type=return_tensors) File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\transformers\feature_extraction_utils.py", line 78, in __init__ self.convert_to_tensors(tensor_type=tensor_type) File "O:\captioning\captioners_clip_interrogator_v2\venv\lib\site-packages\transformers\feature_extraction_utils.py", line 188, in convert_to_tensors raise ValueError( ValueError: Unable to create tensor, you should probably activate padding with 'padding=True' to have batched tensors with the same length.

Furkan Gözükara

thank you so much for such comment. yes i make venv for every installer to not break anything else :)

reaper557

Thank you very much for compiling these models; I am particularly grateful for the auto installs of the venv as every time I tried most manual installs, I would get stuck in a horrible mis-matched dependency loop. Every auto installer I have used from you has worked right out of the box, and you have helped safe me tremendous headaches. :D

Furkan Gözükara

updated to Kosmos-2_v6.zip let me know if that fix your error. also you can email me that particular image that gives error : monstermmorpg@gmail.com

Steve

The above exception was the direct cause of the following exception: Traceback (most recent call last): File "W:\Kosmos\venv\lib\site-packages\gradio\queueing.py", line 495, in call_prediction output = await route_utils.call_process_api( File "W:\Kosmos\venv\lib\site-packages\gradio\route_utils.py", line 231, in call_process_api output = await app.get_blocks().process_api( File "W:\Kosmos\venv\lib\site-packages\gradio\blocks.py", line 1594, in process_api result = await self.call_function( File "W:\Kosmos\venv\lib\site-packages\gradio\blocks.py", line 1176, in call_function prediction = await anyio.to_thread.run_sync( File "W:\Kosmos\venv\lib\site-packages\anyio\to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "W:\Kosmos\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 2134, in run_sync_in_worker_thread return await future File "W:\Kosmos\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 851, in run result = context.run(func, *args) File "W:\Kosmos\venv\lib\site-packages\gradio\utils.py", line 689, in wrapper response = f(*args, **kwargs) File "W:\Kosmos\web_app.py", line 204, in batch_caption_images caption = generate_predictions(img, text_input, is_batch=True) File "W:\Kosmos\web_app.py", line 265, in generate_predictions image_input.save(user_image_path) File "W:\Kosmos\venv\lib\site-packages\PIL\Image.py", line 2439, in save save_handler(self, fp, filename) File "W:\Kosmos\venv\lib\site-packages\PIL\JpegImagePlugin.py", line 653, in _save raise OSError(msg) from e OSError: cannot write mode RGBA as JPEG been getting this error a lot not sure if there is a way to get around it.

Furkan Gözükara

it is running like this : run_pt1.bat , run_pt2.bat , run_pt3.bat run all these 3 with orders and wait each one

Additional Contributions

Doc, I downloaded the LLaVA installer separately and there is no window .bat file, can I only run it on the runpod?

Erik

Cool, I think I'll just use it on hugging face. Everything seems fine and there is no error, but the run_pt3 would just crash at a certain point when you run it. A tutorial comparing them would be great!

Furkan Gözükara

each one has strengths and weaknesses. i plan to compare all later a time. kosmos 2 is working pretty fast with low VRAM. you can try. llava is working perfect. if you can show me entire process how you are running i can point your error. hopefully i will make a tutorial for it

Erik

Which one of these is the best? Also, I installed LLaVA but the program would crash when I get to run_pt3

Furkan Gözükara

updated v7 and fixed the issue. you can either manually downgrade the transformers library or do a fresh install

Furkan Gözükara

same error working on it right now. weird

design master

Hi Doc, I had successfully installed CogVLM_v6 separately (2 days ago, I just needed CogVLM.As a visual designer, lengthy video tutorials are difficult for me, so I followed "runpod_instructions_READ" to perform the installation).but today it no longer works, I neither can open the 7861 port nor get the link after the installation is complete. Error message (KeyError: 'inv_freq')

Furkan Gözükara

i dont think so that error related to batch labeling.

楠陈

CogVLM_v6，After batch labeling, it cannot be used in dreamboth, prompt。Exception training model: ''NoneType' object is not subscriptable'.

Furkan Gözükara

Especially for low VRAM gpus very best. I just added 4 bit 8 bit and 16 bit options

Nenad Kuzmanovic

Kosmos is IMHO the best captioner so far... generated captions requires just a bit of editing...

Furkan Gözükara

no they are all different captioners. you can test all and compare them. hopefully i am working on a massive tutorial for all

Furkan Gözükara

interesting. i have rtx 3060 as my second gpu and when i set cuda visible devices to 1 it works directly and load model into that GPU.

OP-Cast

im a little confused, which one do i run for windows? i installed like it said in the clip interrogator folder, but i downloaded lava and qwen ect, does each do the same thing? running a 3090 and primarily use SDXL for my lora training with Kohya ss

DaOldHeadz Music

I was able to resolve setting the gpu. I have a 3080ti as my primary and a 3090ti as my second. For some reason when selecting #6 (13b Model - Load in 16 bit - 24 GB VRAM) and running LLaVa, I receive a NETWORK ERROR DUE TO HIGH TRAFFIC. PLEASE REGENERATE OR REFRESH THIS PAGE. (error_code: 1). I ran pt1, pt2 then pt3 several times with the same error. When I choose any of the 7B models, it works fine. The 3090ti is at (0 usage) by the way. My monitors are connected to my 3080ti at (0.3 usage).

Furkan Gözükara

I just tested working on my second gpu RTX 3060 : https://pasteboard.co/xYUVhZaoZxoh.png

Furkan Gözükara

you must have changed it inaccurately set CUDA_VISIBLE_DEVICES=1 will change it

DaOldHeadz Music

How do I use gpu1 instead of gpu0. I change set CUDA_VISIBLE_DEVICES=1 in run_pt3 but no luck. Thanks.

Furkan Gözükara

true. 4bit is slower but uses lesser vram.

Furkan Gözükara

just fixed this with v3, it was encoding error thanks for letting me know

Nenad Kuzmanovic

CogLVM rules :-)

Furkan Gözükara

dmn then it is not dependable :/ thanks for letting me know

leem0nchu

Getting this warning when using 4bit. Is this normal behavior? UserWarning: Input type into Linear4bit is torch.float16, but bnb_4bit_compute_type=torch.float32 (default). This will lead to slow inference or training speed. warnings.warn(f'Input type into Linear4bit is torch.float16, but bnb_4bit_compute_type=torch.float32 (default). This will lead to slow inference or training speed.') Over 2 minutes and it never generated a caption. I'm on a 4070 8GB CogLVMv6 I should have pointed out.

Nenad Kuzmanovic

hahha Qwen is so funny and ridiculous, here are some of it's captions: - Sorry, but I can't assist with that. - As an AI language model, I don't have access to images, but I can describe the scene you might be referring to. - As an AI language model, I don't have access to the image you are referring to, so I cannot provide a detailed description of it. However, if you could provide me with more information or context about the image, I would be happy to help you with your query.

Nenad Kuzmanovic

I'm getting this error with batch captioning Qwen: Caption for callisto_ (1).png generated in 1.59 seconds. Estimated time to complete: 348.22 seconds. Processed 14/93. Caption for callisto_ (11).png generated in 6.52 seconds. Estimated time to complete: 356.07 seconds. Processed 15/93. Caption for callisto_ (13).png generated in 3.75 seconds. Estimated time to complete: 347.66 seconds. Traceback (most recent call last): File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\gradio\queueing.py", line 495, in call_prediction output = await route_utils.call_process_api( File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\gradio\route_utils.py", line 230, in call_process_api output = await app.get_blocks().process_api( File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\gradio\blocks.py", line 1590, in process_api result = await self.call_function( File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\gradio\blocks.py", line 1176, in call_function prediction = await anyio.to_thread.run_sync( File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\anyio\to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 2134, in run_sync_in_worker_thread return await future File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 851, in run result = context.run(func, *args) File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\gradio\utils.py", line 678, in wrapper response = f(*args, **kwargs) File "H:\Qwen_captioning\Qwen-VL-Chat.py", line 88, in batch_caption_images caption_file.write(response) File "C:\python3109\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode characters in position 215-217: character maps to Traceback (most recent call last): File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\gradio\queueing.py", line 495, in call_prediction output = await route_utils.call_process_api( File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\gradio\route_utils.py", line 230, in call_process_api output = await app.get_blocks().process_api( File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\gradio\blocks.py", line 1590, in process_api result = await self.call_function( File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\gradio\blocks.py", line 1176, in call_function prediction = await anyio.to_thread.run_sync( File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\anyio\to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 2134, in run_sync_in_worker_thread return await future File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 851, in run result = context.run(func, *args) File "H:\Qwen_captioning\Qwen-VL\venv\lib\site-packages\gradio\utils.py", line 678, in wrapper response = f(*args, **kwargs) File "H:\Qwen_captioning\Qwen-VL-Chat.py", line 88, in batch_caption_images caption_file.write(response) File "C:\python3109\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode characters in position 571-573: character maps to

Furkan Gözükara

you are welcome

leem0nchu

Awesome

Furkan Gözükara

to test those blip1 + clip visions they are inside : captioners_clip_interrogator_v2.zip

Furkan Gözükara

I have added some newer stuff try them. especially blip + clip vision models uses really low VRAM if you have low VRAM.

leem0nchu

I haven’t. I guess I didn’t notice only so the sample scripts on the post a few days ago didn’t that showed 10.5gb vram as lowest

Furkan Gözükara

i am adding low vram options for every script are you testing them? for example 4 bit llava.

leem0nchu

Will there be similar options (I understand not so powerful) for lowvram?

Nenad Kuzmanovic

You are right, but nothing is stopping you to add as much "terms" as you wish... I can do that, but if you can write short instruction which lines in script has to be changed...

Nenad Kuzmanovic

Dots should be replaced with comas, not deleted ;-)

Furkan Gözükara

well sadly this really depends on the user specific input . but i may work on it

Furkan Gözükara

nice info

Nenad Kuzmanovic

I'm testing CogVLM intensively and already i have some patterns for cleaning captions. Like this example, which occurs almost in every caption is: - change "there's a" to "with a" That change mainly concerns the adaptation of the sentence, in the sense of making it as concise as possible and corresponding to the format that SD understands best. Dots "." should be deleted also.

Nenad Kuzmanovic

If you need any help for formating captions, just let me know. It would be great if that script for caption pruning can be integrated in your gradio UI, so we can easily change type of cleaning captions. I am constantly searching for the best caption tools and i am not satisfied with any of them, and if you manage to make it as i suggested, that WILL be gamechanger my friend...

Furkan Gözükara

yes i would also use kohya scripts to improve captions

Nenad Kuzmanovic

I think that can be fixed with custom python script, like in Kohyass/Finetune/clean captions and tags. In that way we can have separate script for style, concept etc.

Furkan Gözükara

thank you for bringing this to my attention. i just changed the default timeout to 60 seconds.

Furkan Gözükara

it is true. they may add unnecessary words. i am also working on Qwen-VL captioning right now to add here

Nenad Kuzmanovic

I edited command and results are quite better: Question: just caption the image with details, colors, items and objects but do not add any description or comment. do not miss any item in the given image Answer: The image showcases a long hallway with white walls and doors. At the end of the hallway, there's a large, ominous portal or gateway. This gateway is surrounded by red, grotesque, and tentacle-like structures. Within the gateway, a massive, menacing face with hollow eyes and sharp teeth is visible. A woman in a long white dress stands in front of the gateway, seemingly confronting or observing the face. The floor is wet, possibly from a recent flood or rain, and there are puddles of water scattered around.

Nenad Kuzmanovic

I've noticed that sentence construction all of VLM's is not very suitable for Stable diffusion training, cause it contains a lot of unnecessary words.. Example: The image showcases a futuristic or sci-fi setting with a central mechanical structure illuminated in green. Within this structure, there's a humanoid figure with a bald head, seemingly in a state of distress or unconsciousness. The environment appears to be a dimly lit room with large windows, possibly suggesting an industrial or laboratory setting. The art style leans towards realism with a touch of surrealism, given the juxtaposition of the mechanical structure and the human figure. Total number of tokens: 99 - 101 (depends of which model is used, gpt-4 says 101). So, undesirable words in this example are: The image showcases, or, Within this structure, possibly suggesting, The art style leans towards realism with a touch of surrealism, given the juxtaposition of the mechanical structure and the human figure. When those words are tokenized, multiple problems potentially arise: 1.Describing the style is not desirable when training the style, but also the character, because the model becomes inflexible and the transfer of the style present in the dataset is not achieved, and character (person) is best to train only with rare token trigger word. This is maybe good for Concept training, but for that we already have LORA and block weights. 2. The number of tokens is unnecessarily inflated, which can be a problem when generating images But this YET has to be tested, I am writing this from the experience I have with training models using filewords

Nenad Kuzmanovic

It is working and yes, IT IS the best captioning model. Amazing how good it is

Joosheen

It is working now, thanks ;) Sometimes there is a problem with error about a network. For others - if u have lower GPU - you need to edit gradio_web_server.py "response = requests.post(worker_addr + "/worker_generate_stream", headers=headers, json=pload, stream=True, timeout=10)" - timeout from 10 to for example 500 and the error will not occur anymore :)

Furkan Gözükara

you were right sorry about that. updated to latest version please download newest versions.

Furkan Gözükara

you were right sorry about that. updated to latest version please download newest versions.

Joosheen

For me llava doesn't work unfortunately :/ I tried to run 6gb vram model. I can open part 1 and part 2 but part 3 is crashing. Gradio is opening but I have several errors: "The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable. warn("The installed version of bitsandbytes was compiled without GPU support. " and "No GPU found. A GPU is needed for quantization". Something is wrong with bitsandbytes?

Furkan Gözükara

you see since you edited cuda to 118 you got the error below

Furkan Gözükara

this is not supposed to happen because it is supposed to install torch 2.2.0 with cu121. but i see it installed 2.2.0 cu118. but my bat file containing this line pip3 install torch==2.2.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121 you see it is installing cu121 also you should install python 3.10.11. but i see you have installed 3.10.9. i have shown this in the video if you want older cuda use this pip3 install torch==2.0.1 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 pip install xformers==0.0.22

Nenad Kuzmanovic

Version 3 was ok, installed easily...

Nenad Kuzmanovic

This last time, i went step by step and installed everything manualy. I edited links with cuda 118 (instead 121)...

Nenad Kuzmanovic

Python is installed in root: C. I dont have antivirus, only Win 10 defender. But i have checked, Kohya installler has installed everything correctly, xformers, bitsandbytes etc....

Nenad Kuzmanovic

ok, i will do it

Nenad Kuzmanovic

Enter your choice (1-2): 1 WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for: PyTorch 2.2.0+cu121 with CUDA 1201 (you have 2.2.0+cu118) Python 3.10.11 (you have 3.10.9) Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers) Memory-efficient attention, SwiGLU, sparse and more won't be available. Set XFORMERS_MORE_DETAILS=1 for more details Traceback (most recent call last): File "H:\CogVLM\web_app_CogVLM.py", line 43, in model = AutoModelForCausalLM.from_pretrained( File "H:\CogVLM\CogVLM\venv\lib\site-packages\transformers\models\auto\auto_factory.py", line 561, in from_pretrained return model_class.from_pretrained( File "H:\CogVLM\CogVLM\venv\lib\site-packages\transformers\modeling_utils.py", line 3032, in from_pretrained raise ImportError( ImportError: Using `load_in_8bit=True` requires Accelerate: `pip install accelerate` and the latest version of bitsandbytes `pip install -i https://test.pypi.org/simple/ bitsandbytes` or `pip install bitsandbytes`. Press any key to continue . . .

Furkan Gözükara

ye 8 bit working nice

Furkan Gözükara

i would do this. uninstall all python cuda. restart computer. install exactly as shown in this video into C drive directly : https://youtu.be/-NjNy7afOQ0 also if you have a antivirus that could be preventing the installation

Nenad Kuzmanovic

I have installed CUDA 11.8 but installer wants to install 12.1. Is that maybe a problem here?

Nenad Kuzmanovic

8bit then

Furkan Gözükara

Hello. Download latest V5. it works with 4bit 8bit and 16bit. i have tested. 16 bit uses more than 30GB VRAM

Furkan Gözükara

uploaded v4. but it requires more than 24gb so i couldn't test yet.

Nenad Kuzmanovic

CogVLM 16 bit.

Furkan Gözükara

you get this error at llava or CogVLM or? CogVLM 8 bit load not working. I am trying to fix. 4 bit load working fine

Nenad Kuzmanovic

Error processing scorn_ (99).png: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument index in method wrapper_CUDA__index_select)

Furkan Gözükara

just added batch. I agree LLaVA better

Nenad Kuzmanovic

That is great, but i found it is not much better than LLAVA, to be honest

Furkan Gözükara

yep currently working on it

Nenad Kuzmanovic

Is it possible to add batch process for CogVLM_v2?

Mike Menders

On average, it reads at 10-12 MB/s speed instead of 100. Sometimes it goes up to 20. As if the model loader reads the model slowly if it is not in the cache. I wrote a PM on Discord.

Furkan Gözükara

ye i also get apex message ignore it. what reading speed do you see on task manager when reading from the disk?

Mike Menders

3TB Toshiba, 7200 RPM with 64 MB cache, 6 Gbit/s, DT01ACA300 Python cache on this HDD (too large for System SSD), and symlinked. And I have this message when start CogVLM: Please 'pip install apex' Apex is installed in venv...

Furkan Gözükara

i think it is related to your disk. the model size is very big and your disk must be very slow

Mike Menders

After I restarted the machine and turned CogVLM back on, it says half an hour loading time! Is this normal? 07:48 - 25:24, 254.01s/it Loading the model is very slow on cold start. If it starts once, then it is fine, then I can turn it off and on, it starts fast (I guess from cache).

Furkan Gözükara

multiple gpu or more powerful gpu

Hassan Alhassan

any way to speed up the LLava captioning? it takes a very long time to caption

Furkan Gözükara

Hello. as I replied your private message it is system error. error of windows. Need more debugging to figure out the reason

耀良梁

Hello! Why do I always get an error when starting the model in the third step when using LLaVA subtitles, such as part 3 - 7b model - 8bit - 8600 MB VRAM.bat, but my graphics card is 3090 and the video memory is enough, my Win11 system directly reports an error Restart, can LLaVA subtitles fail to win the system?

Furkan Gözükara

Hello. updated file names to be more clear you can redownload. the part is the model starting files such as 13b model - 8bit - 15470 MB VRAM.bat

Thomas

I tried to run "13b model - 8bit - 15470 MB VRAM.bat". It said "don't forget to run all run_pt1.bat run_pt2.bat run_pt3.bat" but there's no "run_pt3.bat". Is it a typo?

Furkan Gözükara

yes hopefully i will add that feature very soon just like llava

Furkan Gözükara

yes hopefully i will add that feature very soon just like llava

楠陈

Can it be supported support batch inference

楠陈

CogVLM_v2 Can it support batch marking?

Thomas

Exactly! Thanks

Furkan Gözükara

thank you so much updated the file. you were right. you figured out from pip freeze i included?

Furkan Gözükara

thanks testing right now to fix

Thomas

I only used the files inside your captioners_clip_interrogator.zip, it download the latest bitsandbyte which is currently 0.41.2.post2

Furkan Gözükara

did you upgrade it manually? which script you trying from this post?

Thomas

It worked only after I downgraded bitsandbyte to 0.41.1

Furkan Gözükara

did it still work? i think even though these warnings still should work . i also get a lot of warnings. these happens usually since we use Windows and libraries are optimized only for linux

Thomas

Hello! I can't use this tool because bitsandbytes throw an error : "CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...". I do have the cuda toolkit installed.

Furkan Gözükara

this happens when you skip 1 step. have you seen this video? i have shown there https://youtu.be/ZiUXf_idIR4 13:41 How to use LLaVA for captioning and obtaining prompt ideas and generating more amazing images

gawdman

Hi! Great work and fantastic tutorial. I'm running into an issue however with the LLaVA chatbot. I installed it using your auto installer, but after loading the run bats and the model bat, I get an error "NETWORK ERROR DUE TO HIGH TRAFFIC, PLEASE REGENERATE OR REFRESH THIS PAGE." It seems to work for a second, and then fails. I try refreshing it a number of times and restarting, but no luck. Do you know how to fix this? Thanks again.

RayHell

Batch Processing doesn't work with png files, I need to convert them to jpg and lose a bit of quality.

Furkan Gözükara

great solving

Meito

Yep i just used fast stone to batch convert them to jpg

Furkan Gözükara

i should test thanks for letting me know

Meito

it seems png is affected, i converted them to jpg and it works now

Furkan Gözükara

how to setup is shown here : https://youtu.be/ZiUXf_idIR4 13:41 How to use LLaVA for captioning and obtaining prompt ideas and generating more amazing images

Furkan Gözükara

now this is interesting. you can write a simple python script to convert all your images into png.

Meito

OSError: cannot write mode RGBA as JPEG when doing batch

Meito

how do we setup llava, i'm having trouble running it. which bat files do we run?

Furkan Gözükara

you are welcome. in such cases increasing virtual RAM may help

Guillaume Bieler

thanks, that was the problem :)

Furkan Gözükara

good it means it is starting loading. if failing could be your computer RAM not sufficient.

Furkan Gözükara

Hello. Sadly I don't have information regarding ComfyUI

Guillaume Bieler

Hi, I've had no problem using the app on Linux, but for some reason why I try using blip2-2.7b I get this: Loading caption model blip2-2.7b... Loading checkpoint shards: 0%| | 0/2 [00:00

Dungeon Master

Im trying to use blip captioning to read an image to then use the prompt later in a custom workflow on ComfyUI, as far as iv seen using the WAS node suite I can BLIP Model Loader and BLIP Analyse Image, so apparently I just need add a BLIP model to the correct folder. Im curious if you have any insight into this or tips

楠陈

OK, thank you very much

Furkan Gözükara

it shouldn't be that slow. i also have 3090 working really good. well i suggest you to use LLaVA or Blip2. i think those 2 are best right now for caption

Sarcasticest

Hi. Two questions: 1. I'm trying to use ViT-bigG-14/laion2b_s39b_b160k and blip2-2.7b but it's taking up all GPU vram on my 3090 and taking 2hrs for a single image. Is this right? 2. What is the best CLIP model for sd 1.5?

Furkan Gözükara

hello you were right. i found the error. now added download.py. it will download the gradio app instead of using .bat which was failing in some cases. you can directly run it with python download.py . redownload installer zip file

Furkan Gözükara

yes this works on linux as well. use runpod installer

mypatreonemailacc

Do you have a Linux batch processing script for LLava?

Furkan Gözükara

there is checkout this tweet images : https://twitter.com/GozukaraFurkan/status/1728425482662592625

Furkan Gözükara

true. thanks for comment

DaOldHeadz Music

There should be a path batch processing box at the bottom of the page. I can confirm that batch processing works.

楠陈

Encountered the same problem, but using http://127.0.0.1:7860/ can indeed be used, but there is no place for batch operations and entering folder addresses.

楠陈

It has been automatically installed and enabled, but there is no batch folder input text box. Is it because the automatically installed file is not the latest version? Or you haven't updated it yet, and you have the same problem after reinstalling it several times.

Furkan Gözükara

what have you installed so far? i have shown llava in this video : https://youtu.be/ZiUXf_idIR4 13:41 How to use LLaVA for captioning and obtaining prompt ideas and generating more amazing images hopefully i will make a full tutorial soon

楠陈

How to update and download

Furkan Gözükara

yes my LLaVA gradio already supports batch subtitle given folder images

楠陈

I mean add LLaVA to batch subtitle file

Furkan Gözükara

can you checkout this video? i have shown there quickly : https://youtu.be/ZiUXf_idIR4 13:41 How to use LLaVA for captioning and obtaining prompt ideas and generating more amazing images

DaOldHeadz Music

Hello. When trying to run the URL (http://127.0.0.1:40000) for LLaVA, I get this message from Chrome and Edge. I launched Part 1 then Part 2 then the Model (13b model - 8bit - 15470 MB VRAM). I have a 3090TI. { "detail": "Not Found" } Thanks...

Furkan Gözükara

i am not sure. i think could be but it already uses decent amount of VRAM. if you are meaning batch processing a folder my web app already have that. there is batch process folder input textbox. use it. hopefully will make a tutorial soon

Furkan Gözükara

it works ignore it

楠陈

LLaVA Can it be batch processed?

楠陈

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. llava 1.1.3 requires bitsandbytes==0.41.0, but you have bitsandbytes 0.41.1 which is incompatible.

Furkan Gözükara

1 : if you are doing fine tuning I would compare Blip 2 alone vs LLaVA 2 : for face i don't use any captions. just rare token + class. like ohwx man 3 : try answer 1

Furkan Gözükara

for CPU sadly I don't know :/ but i think LLaVA is also great

So Sha

1. What's the reason we should use the CLIP model alongside the caption model? 2. Do you still believe that ViT-Big-GAN-14 combined with BLIP2-FLAN-T5 offers the best performance for captioning and training a face? 3. I used this combination, but I wasn't satisfied with the results. It added some unrelated and meaningless keywords.

mypatreonemailacc

Which captioning do you recommend for running on the CPU on a laptop? I'm using WD14 right now, which is really fast. Is there anything similar?

Furkan Gözükara

I plan to do a full tutorial today. good reminding me. i should make that

Nenad Kuzmanovic

It would be soo helpfull if u make tutorial of how to change cache folder... Im struggling with free space on my system drive... There is explanation on Hugging face and i tried but something i did wrong and didt manage to make that work.

Furkan Gözükara

so easy. bat files are just pip commands. make your venv and execute them 1 by 1. also you can use runpod_install.sh file which is designed for linux

So Sha

How we can install it on ubuntu ?

Furkan Gözükara

i would try blip 2 + vit bigG 14 but that may not be sufficient. i plan to add llava to the collection as well hopefully soon

Đạt Nguyễn

Is there any suitable for analyzing images with facial expressions? For example, the character is opening his mouth, smiling, closing his mouth, being surprised...

Furkan Gözükara

i prefer currently rare token + class token and not using captions. so like ohwx man or ohwx car or ohwx woman etc. rare token is ohwx and class is the thing you are training.

준영 이

When training a model through Lora, I understood, just as you explained in the video, that there are two methods: instance prompts, class prompts, and the use of captions. Is my understanding correct? If it is, then if I do understand correctly, which of the two methods do you think is better, and could you explain the reasons for your preference?

Furkan Gözükara

i am experimenting with it the effect of captionings. hopefully will make a tutorial but not ready yet

준영 이

Is there a youtube video or something how to use this kind of caption for Lora?

Furkan Gözükara

thanks it may work. i usually make my auto installer

mypatreonemailacc

I found this docker image that should be one-click for LLava in runpod: https://github.com/ashleykleynhans/llava-docker I have not tried it yet, but I will try it soon.

Furkan Gözükara

:D

mypatreonemailacc

Who cares about Windows, everybody uses Linux :)

Furkan Gözükara

i am waiting them to add windows support. after that hopefully will bring it to the runpod and windows both

Furkan Gözükara

yep working amazing

Gerenier

Would it be possible to use llava on a runpod, maybe in batch processing?

Gerenier

Works great with ViT-bigG-14/laion2b_s39b_b160k on 3090 Runpod.

Furkan Gözükara

well i asked them to make it work on windows and they said yes. i am waiting them

mypatreonemailacc

Llava is amazing when it comes to describing a picture: https://llava.hliu.cc It would be great to have a docker image or an installation guide for running it in batch mode. This could be a major game changer when captioning images for training SD models.

Furkan Gözükara

ye pretty useful

EnjoyerOfAll

Hell yeah!

Furkan Gözükara

nice. i am still researching

Furkan Gözükara

thanks but it is outdated not working

mypatreonemailacc

I use WD14 for captioning my datasets for Lora training. If you observe that BLIP2 is better, then let us know!

Furkan Gözükara

I think ViT-bigG-14 + blip2 is great

mypatreonemailacc

If you have a Automatic1111 UI in runpod, you can use these plugins: https://github.com/Tps-F/sd-webui-blip2.git

mypatreonemailacc

BLIP isn't that great, WD14 is already a massive improvement over BLIP. But WD14 doesn't really create full sentences, just a bucket list of what it sees.

Furkan Gözükara

Gradio list 90 models . but i don't know which ones are lighter sadly :( i guess you have to test them

mypatreonemailacc

Do you have the link to the gradio with lightweight models? I think anything less than 10GB should be runnable on a modern laptop CPU.

Furkan Gözükara

Amazing. It is the most powerful one

San Milano

I tested blip2-flan-t5-xxl and it is amazing! It did a great job with images that were very complex

Furkan Gözükara

it is added check out instructions. hopefully a tutorial video coming too

Furkan Gözükara

it uses a lot of VRAM. i think it may on ram. how much RAM you have? by the way I added another captioning gradio which is superb. it also has lightweight models

mypatreonemailacc

Is it possible to run blip2 captioning on my laptop (CPU) ? I can run WD14 from the kohya repository on the cpu, and it's very fast.

Furkan Gözükara

hello i am working on it too. thank you

j

please could you create instructions for Runpod, thanks!

Furkan Gözükara

not yet i am still in research

Daniel Quandt

Do you have any evidence of improvements in Dreambooth training of subjects or styles with captioning? Any comparisons?

Furkan Gözükara

hopefully this one coming today much stronger : https://twitter.com/GozukaraFurkan/status/1712093261249097924 i will update this post

Zsédely Ruben

I tried the 8bit_precision_10_GB_VRAM script, but it's done a poor job: "a woman in a pink skirt and jacket posing"

Furkan Gözükara

i am working on to it as well : https://github.com/haotian-liu/LLaVA/issues/521 also this is coming hopefully tomorrow : https://twitter.com/GozukaraFurkan/status/1711933282529452115

John Dopamine

Hi! This is "JohnDopamine" - Regarding that last question/answer: Would there be anyway (or benefit?) of trying to incorporate "llava" as a method to help caption? Hacksman had mentioned this online demo site: https://llava.hliu.cc/ - which works pretty well if you say "describe this image" and can follow up w/ questions etc. I'm not a coder so don't fully understand what is needed to get anything out of it (if it's even better than the models you have implemented). But I did see there was a commit an hour ago that said training code and dataset have been added. Maybe good news? Thanks for this either way ! code: https://github.com/haotian-liu/LLaVA

Furkan Gözükara

thank you

diffusers

JazakAllah 🙌🏽

Furkan Gözükara

yes they are definitely better. moreover i am working on adding ViT-bigG-14/laion2b_s39b_b160k too to the scripts. it is much different level. both can be compared used and tested