Furkan Gözükara patreon

Kohya brought massive improvements to FLUX LoRA (as low as 4 GB GPUs) and DreamBooth / Fine-Tuning (as low as 6 GB GPUs) training

🕑 Added 2024-11-17 13:45:38 +0000 UTC

Comments

Furkan Gözükara

wow i am gonna test this unexpected

Furkan Gözükara

awesome

Vlad

Thanks. Really, torch 2.5.1dev is slower than torch 2.5

Vicent Guardiola

Oh my god!! It works, I did a fresh installation again, this time without step 2 and the speed improved so much! Now I'n getting 8.60s/it (only waited 100 steps to publish the result). Thank you so much!!

GH

I did a fresh install with 'Windows_Install_Step_1', followed by 'Windows_Update_Kohya_and_Fix_FLUX_Step2'. I completely skipped 'Windows_Install_Torch_2_5_Dev_Huge_Speed_Up' because this step keeps slowing it down for me for whatever reason - i think the 2.5.1 dev torch version in it confuses my setup or something, and the default 2.5 version that gets installed in Step_1 is better for me. Perhaps try doing the same and see if it improves things for you as well

Vicent Guardiola

I'm using the DreamBooth_Tab_Fine_Tuning_Best_FLUX_Configs/24GB_GPU_22900MB_7.3_second_it_Tier_1.json

Vicent Guardiola

I'm only training, I don't use the PC for anything more than training while the process is running.

Furkan Gözükara

ye it is very slow. which config name you are using exactly?

Vicent Guardiola

Hi! With previou configs and install I usually get 10.5s/it aprox, now I'm getting 11.03s/it. I waited 600 steps to record the speed. I made a fresh install with Kohya_FLUX_DreamBooth_v14.zip and now with Kohya_FLUX_DreamBooth_v15.zip and the speed is the same.

Furkan Gözükara

hi Vincent you must have seen some speed improvement with fresh install. can you tell me your step speed? i also have 3090

Furkan Gözükara

awesome ty so much for the feedback

Vicent Guardiola

Hi, have you done a fresh installation? Did you do the installation with Windows_Install_Step_1 and Windows_Install_Torch_2_5_Dev_Huge_Speed_Up and nothing else? I did a fresh installation and it hasn't changed the speed. (I have an 3090) Thanks!

GH

The speed boost is insane!! Gone from 15s/it to 8.5s/it on my 3090, and I can actually use my PC while training as it doesn't hang up anymore! Thanks for updating your scripts, they make my life so much easier!

Furkan Gözükara

ye he decided to reverse after we replied :D

Chris

Seems to be fixed or will be fixed in next pull. Thank you!

Furkan Gözükara

ok he says that now you have to enter yourself max train steps :( this sucks so bad please leave a comment : https://github.com/bmaltais/kohya_ss/issues/2976#issuecomment-2484164299

Chris

Thank you 😊

Furkan Gözükara

currently broken here i opened issue : https://github.com/bmaltais/kohya_ss/issues/2976

Furkan Gözükara

yes as you use it more, it will use more RAM

Furkan Gözükara

yes currently calculations are broken i think. it shows you are doing 534 epoch. not related to config but related to kohya or kohya gui. run installer again and try again please i can test later today

Chris

Has something changed with batch size 7 training, config file or koyha update? batch size 7 training now takes 7 hours with 15 pictures instead of roughly 2 hours on massed compute. i set 15 source pictures, 160 epochs. output gives to much steps and to much epochs for batch size 7? yesterday it worked fine. enable full bf16 training. running training / 学習開始 num examples / サンプル数: 15 num batches per epoch / 1epochのバッチ数: 3 num epochs / epoch数: 534 batch size per device / バッチサイズ: 7 gradient accumulation steps / 勾配を合計するステップ数 = 1 total optimization steps / 学習ステップ数: 1600 steps: 0%| | 0/1600 [00:00

Mikael Svenson

[Edit] figured it out. It was the extra param with --blocks_to_swap 32. Reducing or removing it solved it. [Original] I have 16gb vram and 32gb ram, and for some reason it starts loading data into ram after the "enable full bf16 training" message. Ending with a CUDA out of memory error. I tried both the 16gb lora config and the 12gb lora one for flux. I also allow system managed swap file so it should be able to increase it? Using the v44 installer. enable full bf16 training. Traceback (most recent call last): File "F:\ai\Kohya_GUI_Flux_Installer_v44\kohya_ss\sd-scripts\flux_train_network.py", line 574, in trainer.train(args) File "F:\ai\Kohya_GUI_Flux_Installer_v44\kohya_ss\sd-scripts\train_network.py", line 639, in train unet = self.prepare_unet_with_accelerator(args, accelerator, unet) # accelerator does some magic here File "F:\ai\Kohya_GUI_Flux_Installer_v44\kohya_ss\sd-scripts\flux_train_network.py", line 545, in prepare_unet_with_accelerator accelerator.unwrap_model(flux).prepare_block_swap_before_forward() File "F:\ai\Kohya_GUI_Flux_Installer_v44\kohya_ss\sd-scripts\library\flux_models.py", line 1006, in prepare_block_swap_before_forward self.offloader_single.prepare_block_devices_before_forward(self.single_blocks) File "F:\ai\Kohya_GUI_Flux_Installer_v44\kohya_ss\sd-scripts\library\custom_offloading_utils.py", line 210, in prepare_block_devices_before_forward weighs_to_device(b, "cpu") # make sure weights are on cpu File "F:\ai\Kohya_GUI_Flux_Installer_v44\kohya_ss\sd-scripts\library\custom_offloading_utils.py", line 91, in weighs_to_device module.weight.data = module.weight.data.to(device, non_blocking=True) RuntimeError: CUDA error: out of memory

Luno Lux

Thank you!!

Furkan Gözükara

thanks a lot for the comment

Furkan Gözükara

use attached Fix_Kohya_GUI_Error_Temporary.bat file it will reverse to older working commit. it is error of bmaltais - kohya gui maker

Furkan Gözükara

use attached Fix_Kohya_GUI_Error_Temporary.bat file it will reverse to older working commit. it is error of bmaltais - kohya gui maker

Furkan Gözükara

use attached Fix_Kohya_GUI_Error_Temporary.bat file it will reverse to older working commit. it is error of bmaltais - kohya gui maker

Luno Lux

same here. I also tried with v8 but it creates the same issue. I guess we will have to wait for kohya to fix it (already in issuses on the github page)

Patrick Major

I agree with you @renderartist. I'm a computer engineer full-time and can make everything work on my own just fine. I don't need any of these scripts. BUT... do I feel like spending 30 minutes to an hour to see if Rope is as cool as I think? Not really. Furkan makes the installers so easy I can just spend time playing in the playground I would have spend minutes to hours debugging and setting up.

UC Arcade

same issue. please advise v14

Jeff Penny

I just did a fresh install of Kohya to try this and this is the error I keep getting using the Dreambooth tab, I made sure I selected a Dreambooth config. It happens as soon as I hit start training. Traceback (most recent call last): File "D:\AI\kohya\kohya_ss\venv\lib\site-packages\gradio\queueing.py", line 624, in process_events response = await route_utils.call_process_api( File "D:\AI\kohya\kohya_ss\venv\lib\site-packages\gradio\route_utils.py", line 323, in call_process_api output = await app.get_blocks().process_api( File "D:\AI\kohya\kohya_ss\venv\lib\site-packages\gradio\blocks.py", line 2018, in process_api result = await self.call_function( File "D:\AI\kohya\kohya_ss\venv\lib\site-packages\gradio\blocks.py", line 1567, in call_function prediction = await anyio.to_thread.run_sync( # type: ignore File "D:\AI\kohya\kohya_ss\venv\lib\site-packages\anyio\to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "D:\AI\kohya\kohya_ss\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 2441, in run_sync_in_worker_thread return await future File "D:\AI\kohya\kohya_ss\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 943, in run result = context.run(func, *args) File "D:\AI\kohya\kohya_ss\venv\lib\site-packages\gradio\utils.py", line 846, in wrapper response = f(*args, **kwargs) File "D:\AI\kohya\kohya_ss\kohya_gui\dreambooth_gui.py", line 817, in train_model log.info(max_train_steps_info) UnboundLocalError: local variable 'max_train_steps_info' referenced before assignment

Furkan Gözükara

Thank you so much. Yes mods just deleted my post which was most up voted one :/

renderartist

Thank you for all that you do Furkan, people can be so negative on Reddit...I really don't understand the hate you get, this is all super insightful information. I find so much value in what you share, please keep going! 👍🏼🔥