Furkan Gözükara
patreon
All The LoRA FLUX Training Experiments I Have Done So Far
🕑 Added 2024-09-09 21:00:00 +0000 UTCI have been using a machine on RunPod that has 8x RTX A6000 over a week now. I have done all the following experiments with each one being 3000 steps.
Latest configs shared on : https://www.patreon.com/posts/kohya-flux-lora-110879657
All findings, grids, and best configs and 1-click installers for Windows, RunPod and Massed Compute shared here : https://www.patreon.com/posts/kohya-flux-lora-110293257
Best configs got updated and total number of trainings reached to 104
You can see saved safetensors files in below repos. Click load more button at the end to see entire repo files
https://huggingface.co/MonsterMMORPG/FLUX_Kohya_SS_Massive_Research/tree/main
https://huggingface.co/MonsterMMORPG/FLUX_Kohya_SS_Massive_Research_Part2/tree/main
https://huggingface.co/MonsterMMORPG/FLUX_Kohya_SS_Massive_Research_Part3/tree/main
https://huggingface.co/MonsterMMORPG/FLUX_Kohya_SS_Massive_Research_Part4/tree/main
https://huggingface.co/datasets/MonsterMMORPG/FLUX_Kohya_SS_Massive_Research_Part5/tree/main
https://huggingface.co/MonsterMMORPG/FLUX_Kohya_SS_Massive_Research_Part6_Final/tree/main
https://huggingface.co/MonsterMMORPG/FLUX_Kohya_SS_Massive_Research_Part6_T5_XXL/tree/main
All Tested Experiments List So Far
1) 16GB_Config.safetensors
2) 1e_04_bf16_128_rank-000170.safetensors
3) 1e_04_bf16_128_rank_full_bf16.safetensors
4) 1e_04_fp16_128_rank-000180.safetensors
5) 1e_04_fp16_128_rank_full_fp16-000180.safetensors
6) 1e_04_fp8_32_rank-000170.safetensors
7) 1e_04_fp8_bf16_accelerate_32_rank-000160.safetensors
8) 1e_04_fp8_bf16_accelerate_full_bf16_32_rank-000160.safetensors
9) 1e_04_fp8_full_fp16_32_rank-000170.safetensors
10) 2e_04_fp8_bf16_accelerate_full_bf16_32_rank.safetensors
11) 3e_04_fp8_bf16_accelerate_full_bf16_32_rank.safetensors
12) 6e_05_best_additive_sigma.safetensors
13) 6e_05_best_additive_uniform.safetensors
14) 6e_05_best_max_grad_norm_0.safetensors
15) 6e_05_best_raw_sigma.safetensors
16) 6e_05_best_raw_sigmoid.safetensors
17) 6e_05_best_sigma_scaled_sigma.safetensors
18) 6e_05_best_with_reg_images.safetensors
19) 6e_05_bf16_128_rank_full_bf16.safetensors
20) 7e_05_bf16_128_rank_full_bf16.safetensors
21) 8e_05_bf16_128_rank_full_bf16.safetensors
22) 8e_05_fp8_bf16_accelerate_full_bf16_32_rank.safetensors
23) 9e_05_bf16_128_rank_full_bf16.safetensors
24) 9e_05_fp8_bf16_accelerate_full_bf16_32_rank.safetensors
25) Best_V4_T5_1e_05-000150.safetensors
26) Best_V4_T5_1e_05_TE_5e_05-000150.safetensors
27) Best_V4_T5_2e_05-000150.safetensors
28) Best_V4_T5_3e_05-000150.safetensors
29) Best_V4_T5_4e_05-000150.safetensors
30) Best_V4_T5_5e_05-000150.safetensors
31) Best_V4_T5_5e_05_With_Captions-000150.safetensors
32) Best_v1.safetensors
33) Best_v1_3e_5.safetensors
34) Best_v1_4e_5.safetensors
35) Best_v1_512px.safetensors
36) Best_v1_512px_Lowest_VRAM.safetensors
37) Best_v1_5e_5.safetensors
38) Best_v1_5e_5_max_grad_norm_0.safetensors
39) Best_v1_Lowest_VRAM.safetensors
40) Best_v1_T5_attention_mask.safetensors
41) Best_v1_cfg_3_5.safetensors
42) Best_v1_max_grad_norm_0.safetensors
43) Best_v1_reg_0_1_percent.safetensors
44) Best_v1_reg_prior_lost_0_3.safetensors
45) Best_v1_reg_prior_lost_0_6.safetensors
46) Best_v1_reg_prior_lost_1_0_apply_t5_attention_mask.safetensors
47) Best_v2.safetensors
48) Best_v2_1280.safetensors
49) Best_v2_1280_256rank.safetensors
50) Best_v2_1536.safetensors
51) Best_v2_1536_256rank.safetensors
52) Best_v2_2e_05_Clip_1e_04.safetensors
53) Best_v2_2e_05_Clip_3e_04.safetensors
54) Best_v2_2e_05_Clip_6e_05.safetensors
55) Best_v2_2e_05_Clip_8e_05.safetensors
56) Best_v2_4e_05_Clip_1e_04.safetensors
57) Best_v2_4e_05_Clip_3e_04.safetensors
58) Best_v2_4e_05_Clip_6e_05.safetensors
59) Best_v2_4e_05_Clip_8e_05.safetensors
60) Best_v2_5e_05_Clip_1e_05.safetensors
61) Best_v2_5e_05_Clip_5e_05.safetensors
62) Best_v2_FP8.safetensors
63) Best_v2_FP8_LowVRAM.safetensors
64) Best_v2_Flow_Shift_0_5.safetensors
65) Best_v2_Flow_Shift_1.safetensors
66) Best_v2_Rank16.safetensors
67) Best_v2_Rank256.safetensors
68) Best_v2_Rank32.safetensors
69) Best_v2_Rank4.safetensors
70) Best_v2_Rank64.safetensors
71) Best_v2_Rank8.safetensors
72) Best_v2_Shift_Sample_3e_04.safetensors
73) Best_v2_Shift_Sample_4e_04.safetensors
74) Best_v2_Shift_Sample_6e_04.safetensors
75) Best_v2_Shift_Sample_7e_04.safetensors
76) Best_v2_Split_QKV.safetensors
77) Best_v2_joycaption.safetensors
78) Best_v2_joycaption_Split_QKV.safetensors
79) Best_v2_only_OHWX.safetensors
80) Best_v2_shift.safetensors
81) Best_v2_shift_1e_05.safetensors
82) Best_v2_shift_2e_05.safetensors
83) Best_v2_shift_3e_05.safetensors
84) Best_v2_shift_4e_05.safetensors
85) Best_v2_shift_6e_05.safetensors
86) Best_v2_shift_9e_06.safetensors
87) Best_v2_shift_value_1.safetensors
88) Best_v3.safetensors
89) Best_v3_4e_05_TE_6e_04.safetensors
90) Best_v3_Clip_Skip_0.safetensors
91) Best_v3_Clip_Skip_2.safetensors
92) Best_v3_Discrete_Flow_1_0.safetensors
93) Best_v3_Discrete_Flow_3_0.safetensors
94) Best_v3_TE_0.safetensors
95) Best_v3_xFormers.safetensors
96) Best_v4.safetensors
97) Best_v4_Clip_Skip_1.safetensors
98) CivitAI_FLUX.safetensors
99) Rank_5_11700MB_14_62_Second_IT.safetensors
100) Rank_8_15300MB_4_53_Second_IT.safetensors
101) Rank_8_15400MB_3_81_Second_IT.safetensors
102) Rank_8_15450MB_3_80_Second_IT.safetensors
103) Rank_8_15600MB_4_89_Second_IT.safetensors
104) Rank_8_9500MB_10_00_Second_IT.safetensors
Comments
Furkan Gözükara
yes these are my personal trainings not open to download
锦涛 何
Hello brother, I can't download the configuration file on huggingface. I need the author's consent. My name is he, a purple avatar. Please agree to my download request.
Furkan Gözükara
go for rank 4 it should be really fast , load fp16 version of the model it will auto cast
starysmok
Which configuration would you recommend for training a character model on an RTX 4090 so that it doesn’t take all day? My model is FP8 PixelWave
Furkan Gözükara
ye these are quality ranks :D
Raul Ciotescu
oh ... now i get it ... those are your ranks ... not lora ranks :D
Raul Ciotescu
what would be the explanation behind rank 1 being the best? it trains less params then rank 2?
Furkan Gözükara
yes totally normal. currently i am getting 6.19 second it on 8x gpu with 4x gpu fast rank
Lukas
I use your 4x_GPU_Rank 1, it shows 2500 steps, 193 epoch and 9.24s/it. I thougt it will be faster. But does it sound right for you?
Lukas
Thanks! I will share my results
Furkan Gözükara
for runpod go with Rank 1 - it is best - if it be slow go with rank 2
Lukas
So which config ist the best for style? I will give it a try in runpod
Furkan Gözükara
You are welcome
Mark Sutherland
Thank you
Furkan Gözükara
kohya gui doesn't know automatically split so we set it manually. when you pay attention to the cmd window you will see it shows total epoch. it wasnt exactly 200 in my case but it was 188 total epoch on CMD when i set 50 for 15 images. probably due to i dont have 16 images :) or 20 - not dividable to 4
juggzz143
Thanks for the update... Dividing the number of epochs by the number of gpu's was the key piece of info that I needed. I assumed that was what I needed to do but I wasn't sure and didn't want to waste time if it was going to cause it to be undertrained.
Furkan Gözükara
just added configs you get effective 1.4 second / it check the updated post : https://www.patreon.com/posts/110293257 you can even get better with batch size 2 and 4x but i dont suggest due to quality loss
Furkan Gözükara
give me few minutes to add
juggzz143
Yes pls I am currently getting 3.85s/it on my 4090 but 9.10s/it on Massed Compute with 4 A6000s.
Furkan Gözükara
good idea i can share massed compute max speed
juggzz143
Can you share the settings to make kohya_ss train faster with multiple gpu's? I am currently training with 4 A6000's on Massed Compute but it seems slower than my actual 4090.
Furkan Gözükara
it will help me for sure. after lora hopefully full fine tuning / dreambooth next
Furkan Gözükara
yes i have : FLUX Auto Model Downloader for Windows, RunPod and Massed Compute https://www.patreon.com/posts/110323512 zip file also has auto downloader for training needed models : https://www.patreon.com/posts/kohya-flux-lora-110293257
Mark Sutherland
Do you have a one click installer for flux which downloads the models etc cheers
Ec Jep
Thanks for all of your work and research. Incredibly helpful! Does your LORA research help you determine what DB fine tuning training will work best?
