Furkan Gözükara
patreon
Kohya FLUX LoRA and Fine Tuning Training Full Tutorial For Local Windows and Cloud RunPod and Massed Compute for Research & Development
🕑 Added 2024-08-30 10:00:00 +0000 UTCI will update this page as I progress. I am just starting so I will add the info. Nothing ready yet
FOLLOW THIS THREAD NOW FOR TUTORIAL : https://www.patreon.com/posts/110879657
Who has fbgemm.dll error on Windows 11 the solutions posted below topic
Quick solution is as below (verified DLL)
Download this dll and copy it to your /windows/system32 folder. https://www.dllme.com/dll/files/libomp140_x86_64/00637fe34a6043031c9ae4c6cf0a891d/download
Latest Zip file here : Kohya_GUI_Flux_Installer_27.zip
30 August 2024
Clip Large training research still in work
I have trained FLUX Schnell Turbo model and surprisingly it worked however results are not great
You can see full grid here : Flux_Schnell_Turbo_Test.jpg
I also trained with 4x GPU, used our config and enabled T5 masked attention too and the results are amazing
Almost linear speed gain
You can see full grid here : 4x_GPU_Results.jpg
28 August 2024
I have started training Clip L text encoder
Early experiments are promising I am going to do full research and update configs hopefully
Here first comparison grid : Clip_L_Text_Encoder_Training_Early_Experiment.jpg
27 August 2024 Update V2
Test prompts improved and moved into a folder
New Best_Configs_Better_Colors folder added
The difference is using Timestep Sampling as shift and using Discrete Flow Shift value as 1
I will do more research about this
You can see quick research comparison grid here : Shift_Value_1_Test.jpg
Shift timestep sampling definitely improved colors and overall composition
In some images you will see reduced likeness but in some images even better
It requires more research
Clip L training arrived (smaller text encoder)
Both Clip-L training and Timestep Shift sampling experiments are being done right now
27 August 2024 Update
Batch size experiments section added and config for 4x GPU added
Configs are inside Best_Configs folder inside above zip file
34 Distinct unique prompts for testing added to the zip file
Hopefully I am going to test time step shift more and Clip L Text Encoder training (Kohya just announced)
Newest Configs :
Rank 1 > Rank 2 > Rank 3 and so on - quality assessment
The quality difference between Rank 1 to Rank 6 are not very much but starting from Rank 7 the quality will degrade since we reduce training resolution and LoRA rank
So you can pick faster over slower if you are in hurry
These are early steps speeds (like first 50 and Kohya doesn't display latest step speed) so as you train more you will get like at least 25% faster speeds
These are raw VRAM usages so you need to look your VRAM usage before starting training
The tests are made on Linux on Massed Compute on RTX A6000 GPU - 1024x1024 resolution - 128 LoRA rank
Rank_1_28700MB_Slow.json - 16bit - 8.53 second / it
Rank_2_27360MB_Fast.json - 16bit - 4.49 second / it
Rank_3_18246MB_Slow.json - 8bit - 8.62 second / it
Rank_4_16960MB_Fast.json - 8bit - 4.61 second / it
Rank_5_11498MB_Slow.json - 8bit - Single Layers - 12.12 second / it
Rank_6_10222MB_Fast.json - 8bit - Single Layers - 9.42 second / it
Rank_7_9502MB.json - 8bit - Single Layers - 8.61 second / it - 896px
Rank_8_15406MB.json - 8bit - 3.71 second / it - 64 LoRA Rank - 896px
Rank_9_7514MB.json - 8bit - Single Layers - 5.8 second / it - 64 LoRA Rank - 512px
My suggestions for GPUs
8 GB GPUs : Rank_9_7514MB.json
10 GB GPUs : Rank_7_9502MB.json
12 GB GPUs : Rank_5_11498MB_Slow.json
16 GB GPUs : Rank_5_11498MB_Slow.json - if you need speed : Rank_8_15406MB.json
24 GB GPUs : Rank_3_18246MB_Slow.json
48 GB GPUs : Rank_1_28700MB_Slow.json
Batch Size Experiments And Multi GPU Usage
I have used Rank_2_27360MB_Fast.json config to test batch size impact on RTX A6000
The speed gain from batch size is almost none thus I don't suggest since you lose quality
Lesser batch size = better quality
Only Batch size 2 gives you some gain so you may use it if you wish
Batch size 1 : 4.54 second / it : effective speed same
Batch size 2 : 7.98 second / it : effective speed per step 3.99 second / it
Batch size 3 : 12.43 second / it : effective speed per step 4.14 second / it
Batch size 4 : 15.28 second / it : effective speed per step 3.82 second / it
Batch size 5 : 20.18 second / it : effective speed per step 4.03 second / it
Therefore I have added batch size 1 but 4 A6000 GPU Speed config : 4x_GPU_Batch_Size_1.json
With this config you get 5.75 second / it and effective speed per step is : 1.4375 second / it
When you use multiple GPU you need to divide epoch count to number of GPUs
So 200 epoch becomes 50 for 4x GPU
Also increase LR with this formula - best LR x (batch size * number of GPUs / 2) so in this case 0.00005 * (4/2) = 0.0001
So if you make batch size 2 it becomes = 0.00005 x (2x4/2) = 0.0002
The zip file now has 4x_GPU_Batch_Size_1.json and 4x_GPU_Batch_Size_2.json
I suggest you to use 4x_GPU_Batch_Size_1.json on a 4x RTX A6000 GPU machine
Inference Config for Evaluating Results of Experiments
I use FP16 FLUX dev model in 16 bit precision with UniPC sampler and 30 steps and SwarmUI grid system
You Don't Necessarily Need Class Prompt With FLUX
In the below grid images
trained with ohwx man : model name > Best_v2-000150 - first column
with only ohwx : model name > Best_v2_only_OHWX-000150 - second column
And then generated images with
prompt having ohwx man > More_Time_Shift_Test_and_Only_Ohwx_Token_Tested_and_Ohwx_Man_Used_in_Prompt.jpg
prompt only have ohwx > More_Time_Shift_Test_and_Only_Ohwx_Token_Tested_and_Only_Ohwx_Word.jpg
prompt only have man > More_Time_Shift_Test_and_Only_Ohwx_Token_Tested_and_Only_Man_Word.jpg
I predict that the T5 Text Encoder already encode image in a way that it fully tokenize it. So even if we don't define captions, our images are fully tokenized
This also results in overtraining in our case
Hopefully I will try lower LR and Clip-L training to solve this issue and I will try fine-tuning as well
So if you are going to train a 2 same class like 2 man in same training you can just caption them as ohwx and bbuk and such and try it
I still find ohwx man slightly better than ohwx
Also full captions reduces likeliness of a person training
Different LoRA Ranks Experiments
Full LoRA Rank impact : https://huggingface.co/MonsterMMORPG/Generative-AI/resolve/main/Full%20LoRA%20Rank%20Impact.jpg
Higher LoRA rank learns more details but overfits model more. I like 128 but it is personal. Look grid above for full impact
Higher Resolution Training Impact
I have done trainings at 1024x1024 (default), 1280x1280 and 1536x1536px
Then I have generated images in 1024x1024, 1280x1280, 1536x1536 and 1920x1080px for each training
Higher resolution training slightly improves quality but not dramatic probably doesn't worth the speed loss it causes
Full grid results are below
High Res Training Comparison 1024px.jpg , High Res Training Comparison 1280px Part 1.jpg , High Res Training Comparison 1536px Part 1.jpg , High Res Training Comparison 1536px Part 2.jpg
QKV Split Attention and JoyCaption Detailed Captioning Trainings Results
I have comprehensively tested QKV split attention and detailed captioned dataset trainings
I don't find split QKV brings benefit and brings huge speed loss
For person training detailed captions reduced the resemblance and didn't improve and flexibility or quality. But the resemblance loss was way lower than SDXL
So my conclusions are do not use split QKV and detailed captions for training a person
For style training probably detailed Joy captions will work great - i should test it too
Full grid results are below
QKV Split and JoyCaption Result 1024px.jpg , QKV Split and JoyCaption Result 1920px.jpg
Time Step Sampling Shift Experiments
This one hugely improved stylization capability of the model but reduced resemblance and the realism
I plan to hopefully do more research on this - 8 more trainings
Wait for them. Full grids below
Apply T5 Attention Mask and detailed Regularization / Classifications Images Impact Experiments
I find that Apply T5 Attention Mask improves quality but the tradeoff is slightly improved VRAM and some serious speed los
I also have compared my training workflow with default CivitAI FLUX LoRA training and ours much better
Regularization / classification images didn't improve results with none of the tested configs - I tested so many - check grids to see in full detai
Full grids are shown below
Reg_T5_Attention_Mask_CivitAI_50_Epoch.jpg, Reg_T5_Attention_Mask_CivitAI_100_Epoch.jpg, Reg_T5_Attention_Mask_CivitAI_150_Epoch.jpg, T5_Attention_Mask_v1.jpg, Civit_Reg_15GB_150_Epoch_Compare.jpg
More Different Experiments Are Historically shared below read the entire thread
-
24 August 2024 Update V2
Windows_Download_Training_Model_Files.bat file added to download necessary training model files into the bat file run directory
A 1-min video published to show how to set Accelerate for Kohya SS GUI : https://youtu.be/adVhm9aI9Gc
This set fixes caching stuck problem
24 August 2024 Update
Massed Compute installer fixed and super simplified just 1 step and auto opens browser
Massed compute instructions updated and auto models downloader added - downloads all necessary models automatically Dev FP16, T5 FP16, VAE and Clip
Fix_For_FLUX_Step_2.bat improved - don't forget to run this for lower VRAM usage and 10GB Config
10 GB config updated for latest GUI
Solution for KeyError: 'time_embed.0.weight' error
When you loaded the config make sure that this is selected
23 August 2024 Update
Massed Compute and RunPod Installers fully added (read Massed_Compute_Kohya_FLUX_Instructions and RunPod_Install_Instructions)
16_GB_Config_May_Be_Lower_Quality_In_Test added - still in test - changes are rank 32 and 896 pixel resolution training but super fast compared to lowest VRAM
Download newest Kohya_GUI_Flux_Installer_v13.zip and extract into any folder
48_GB_GPUs_v2 (27 GB VRAM) and 24_GB_GPUs_v2 (17 GB VRAM) yields almost same quality
10_12_16GB_GPUs_v2.json (yields almost same quality as best ones but 3-5 times slower due to VRAM optimizations) - uses 10183 MB peak so if uses shared VRAM and be too slow reduce LoRA Rank or resolution to like 896x896 - I am testing LoRA rank impact at the moment
16_GB_Config_Slightly_Lower_Than_10GB_But_5_Times_Faster.json yields lower quality than 10_12_16GB_GPUs_v2.json but 3-5 times faster than it
Newest test Grids are as below
Reg_T5_Attention_Mask_CivitAI_50_Epoch.jpg, Reg_T5_Attention_Mask_CivitAI_100_Epoch.jpg, Reg_T5_Attention_Mask_CivitAI_150_Epoch.jpg, T5_Attention_Mask_v1.jpg, Civit_Reg_15GB_150_Epoch_Compare.jpg
Tested all configs file updated to V5 : Tested_All_Configs_V6.zip
So what are the newest changes? I have enabled T5 Attention Mask which I believe slightly improves results
T5 Attention Mask Tradeoffs
However T5 Attention Mask increases VRAM usage by 1 GB for 16gb, 24gb and 48 gb configs, thus I didn't enable it for 16 GB config. Now 48 GB config uses 28 GB, 24 GB config uses 18 GB with attention masking
Moreover, T5 Attention Mask slows down the training significantly so that is another tradeoff
What does it do explained here : https://poe.com/s/EHIviAdVuZds5XBGDYWP
What is next? I am going to test quality impact of different LoRA network ranks next 4, 8, 16, 32, 64, 128
16 GB Config yielded slightly worse results than 10 GB config probably due to reduced resolution and LoRA network rank
CivitAI default settings training performed worse than our very best config with exactly same dataset and caption
Regularization / classification images tests failed. Still yields non-resemblance images. I don't see its benefit at this point. Tested with different Prior loss weight parameters but non yielded better results
Windows training is still significantly slower than Linux training. I am still researching a fix for this issue. Asked on Pytorch and Diffusers GitHubs
Full Kohya SS GUI setup screenshots added to the installer zip file
22 August 2024 Update
How to setup Kohya interface for FLUX full screenshot : Example Full Setup.jpg
FLUX LoRA training almost perfected
Use LoRA tab to load configs not DreamBooth
I also made 1 click Kohya SS GUI installer for FLUX
Run Windows_Install_Step_1.bat - select option 1 and then exit with option 7 once completed - no need other options
Run Fix_For_FLUX_Step_2.bat and then use Windows_Start_Kohya_SS.bat to start
Step 2 will improve training speed like 100% and VRAM usage like 25% with just proper library upgrade
Same upgrade added for Massed Compute read the
You are ready to use FLUX training with most performant way now
From newest configs I feel like Best_v1_5e_5_max_grad_norm_0 is the best
But Best_v1_5e_5 is also very good. There is like some lightning difference
When you extract the above zip file you will see configs as
48_GB_GPUs.json for 48 GB GPUs
24_GB_GPUs.json for 24 GB GPUs - uses around 17-18 gb with newest libraries
16GB_Compare_Speed_With_10_12_16_Config.json - If you have 16 GB try this and compare speed with 10_12_16GB_GPUs.json
10_12_16GB_GPUs.json - For 10, 12, 16 GB GPUs. If you have 10 GB and be very slow you can reduce LoRA Rank and also resolution to 512,512
512px training yields inferior quality and VRAM usage almost same - but 2.5x speed
New tested configs full quality grids : 50 epoch (750 steps) , 100 epoch (1500 steps) , 150 epoch (2250 steps)
For 24 GB Config RTX 3090 Speed should be close to 4 second / it - 1024x1024px
For 10_12_16 GB config speed drops like 3 to 5 times due to full optimizations
Training dataset 15 images, 1 repeat, train like 200 epochs
22 August 2024 Update
Section "FLUX Huge Sampler + Scheduler Test For a Very Hard Prompt" added to the post
Section "FLUX Guidance Scale Grid Test on LoRA" added to the post
Newests tests completed and new best configs uploaded as
Surprise that now wen have 10 GB config (LoRA rank 128) - 10 GB config will be published hopefully in 8-9 hours
If you have 10 GB GPU try to reduce LoRA rank until it works better also minimize your VRAM usage before starting the training
So the current best configs are like this
For 48 GB GPU : 6e_05_best_raw_sigmoid.json
For 24 GB GPU : 6e_05_best_raw_sigmoid_24GB.json,
For 10, 12, 16 GB GPUs : lowest_vram.json
For lowest VRAM config to work you have to activate VENV of Kohya GUI and execute below commands 1 by 1
This below thing will speed up training almost 100% and reduce VRAM usage greatly but results unknown yet (training atm). You can also do that if you have 24 GB GPUs like 4090
First update Kohya to latest, run 1 time, then activate venv and install below, then edit gui.bat file and comment or remove python.exe .\setup\validate_requirements.py and if %errorlevel% neq 0 exit /b %errorlevel% to prevent auto reinstall requirements
pip install torch==2.4.0+cu124 --index-url https://download.pytorch.org/whl/cu124
pip install torchvision==0.19.0+cu124 --index-url https://download.pytorch.org/whl/cu124
21 August 2024 Update
So far 16 different tests completed
For 48GB GPUs use 6e_05_bf16_128_rank_full_bf16.json - train up to 200 epochs and compare checkpoints
For 24GB GPUs use 6e_05_fp8_bf16_accelerate_full_bf16_32_rank.json - train up to 200 epochs and compare
I will try to get better results and reduce number of necessary epochs
I will like reg images would help but I haven't tested yet
20 August 2024 Update
So far tested configs added to the attachment and more results added to the post
More configs are on training
Windows Requirements
Python 3.10, FFmpeg, Cuda 11.8, C++ tools and Git
If it doesn't work make sure to below tutorial and install everything exactly as shown in this below tutorial
How To Install
I am going to use GUI version of the Kohya
Currently It doesn't have FLUX at the main branch
So clone it into a new folder like c:/flux_train
Before installing open a cmd
Type : git checkout sd3-flux.1
Then install as usual
FLUX Training Discussions With Lots of Info
Models Links Downloads
FLUX dev FP16 (23.8 GB) : https://huggingface.co/OwlMaster/realgg/resolve/main/flux1-dev.safetensors
Download Clip L (250 MB) : https://huggingface.co/comfyanonymous/flux_text_encoders/resolve/main/clip_l.safetensors
T5 XXL FP16 (9.8 GB) : https://huggingface.co/comfyanonymous/flux_text_encoders/resolve/main/t5xxl_fp16.safetensors
FLUX VAE (335 MB) : https://huggingface.co/OwlMaster/realgg/resolve/main/ae.safetensors
FLUX Fine Tuning Lower VRAM Optimizations
--optimizer_type adafactor --optimizer_args "relative_step=False" "scale_parameter=False" "warmup_init=False"
--timestep_sampling sigmoid --model_prediction_type raw --guidance_scale 1.0
--blockwise_fused_optimizers --double_blocks_to_swap 6 --cpu_offload_checkpointing
Options are almost the same as LoRA training. The difference is `--blockwise_fused_optimizers`, `--double_blocks_to_swap` and `--cpu_offload_checkpointing`. `--single_blocks_to_swap` is also available.
`--blockwise_fused_optimizers` enables the fusing of the optimizer for each block. This is similar to `--fused_backward_pass`. Any optimizer can be used, but Adafactor is recommended for memory efficiency. `--fused_optimizer_groups` is deprecated due to the addition of this option for FLUX.1 training.
`--double_blocks_to_swap` and `--single_blocks_to_swap` are the number of double blocks and single blocks to swap. The default is None (no swap). These options must be combined with `--blockwise_fused_optimizers`.
`--cpu_offload_checkpointing` is to offload the gradient checkpointing to CPU. This reduces about 2GB of VRAM usage.
--network_args train_blocks=single - reduces vram
All these options are experimental and may change in the future.
The increasing the number of blocks to swap may reduce the memory usage, but the training speed will be slower. `--cpu_offload_checkpointing` also slows down the training.
Swap 6 double blocks and use cpu offload checkpointing may be a good starting point. Please try different settings according to VRAM usage and training speed.
Previous Master Kohya Tutorials
How To Do SDXL LoRA Training On RunPod With Kohya SS GUI Trainer & Use LoRAs With Automatic1111 UI
Become A Master Of SDXL Training With Kohya SS LoRAs - Combine Power Of Automatic1111 & SDXL LoRAs
Massed Compute
Massed compute A100 discount : A100_2ybG1e9StAdA
Auxiliary Scripts and Tutorials
Learn how to use SwarmUI : https://youtu.be/HKX8_F1Er_w
Learn how to use SwarmUI on Cloud (RunPod, Massed Compute, Kaggle) : https://youtu.be/XFUZof6Skkw
Learn how to use FLUX with SwarmUI : https://youtu.be/bupRePUOA18
Learn how to download models from Hugging Face and CivitAI and how to upload Hugging Face for backup : https://youtu.be/X5WVZ0NMaTg
FLUX Guidance Scale Grid Test on LoRA
As you know FLUX has CFG set as 1
FLUX dev model has FLUX guidance scale
You can download full grid test from this link and check out
I feel like FLUX Guidance 4.0 is best but you can push more to get more accurate prompt following
FLUX Huge Sampler + Scheduler Test For a Very Hard Prompt
Prompt is : A cinematic shot featuring a photo-realistic image of an Ohwx man riding an enormous and terrifying Tyrannosaurus rex in the heart of Jurassic Park. The scene is set against a backdrop of dense, prehistoric jungle with towering trees and lush greenery, creating an intense and thrilling atmosphere. The T-rex is depicted in all its majestic and fearsome glory, with its massive jaws open wide, sharp teeth glistening, and powerful muscles rippling as it charges forward. The Ohwx man, in contrast, is shown with a determined and fearless expression as he expertly rides the gigantic predator, his posture exuding confidence and control. The image is captured in a highly cinematic style, with dramatic lighting and dynamic angles that emphasize the scale and intensity of the scene. The focus is partially directed towards the face of the Ohwx man, ensuring that his features are clearly visible and convey a sense of resolve and bravery, while maintaining the overall composition of the dynamic action scene.<segment:face,0.7>photo of ohwx man
So far my findings are as below as promising sampler + scheduler
Full grid link : click here
euler + normal - default
euler + karras - cinematic look
heun + karras , heunpp2 + karras
dpm_2 + karras or exponential
dpm_fast + normal
lcm + normal or karras : cartoon look
unipc + normal or karras
I did bigger tests and my conclusion is that sadly Karras scheduler which makes it more realistic is only useable with DMP_2 and even that is not great
Also I think best sampler is UniPC
Below all test results
LCM_Normal_vs_Karras.jpg , Eular_A_Normal_vs_UniPC_Normal.jpg , Eular_Normal_vs_Karras.jpg , UniPC_Normal_vs_Karras.jpg , DPM_2_Normal_vs_Karras.jpg
Experiments and VRAM Usages
A6000
Accelerate launch Mixed precision BF16 - highvram - 128 rank : 44.18 GB - 8.25 second / it
Accelerate launch Mixed precision BF16 - full BF16 - highvram 128 rank : 42.58 GB - 8.25 second / it
Accelerate launch Mixed precision BF16 - lowvram - 128 rank : 44.18 GB - 8.25 second / it : 44.18 GB - 8.25 second / it
Accelerate launch Mixed precision BF16 - full BF16 - lowvram 128 rank : 42.58 GB - 8.25 second / it
Accelerate launch Mixed precision FP16 - highvram - 128 rank : 44.14 GB - 7.83 second / it
Accelerate launch Mixed precision FP16 - full FP16 - highvram - 128 rank : 42.55 GB - 7.83 second / it
Accelerate launch Mixed precision BF16 - highvram - 128 rank - Memory efficient attention : 44.18 GB - 8.25 second / it
Accelerate launch Mixed precision BF16 - highvram - 128 rank - Split Mode : error
Accelerate launch Mixed precision BF16 - highvram - 128 rank - Split Mode train blocks double : error
Accelerate launch Mixed precision BF16 - highvram - 128 rank - Split Mode train blocks single : error
Accelerate launch Mixed precision FP16 - fp8 base - highvram - 128 rank : 33.53 GB - 7.99 second / it
Accelerate launch Mixed precision FP8 - fp8 base - highvram - 128 rank : error
Accelerate launch Mixed precision FP8 - fp8 base - full FP16 - highvram - 128 rank : error
Accelerate launch Mixed precision FP16 - fp8 base - highvram - full FP16 - 128 rank : 24.39 GB - 7.96 second / it
Accelerate launch Mixed precision BF16 - fp8 base - highvram - 128 rank : 33.55 GB - 8.38 second / it
Accelerate launch Mixed precision BF16 - fp8 base - full BF16 - highvram - 128 rank : 24.4 GB - 8.35 second / it
Accelerate launch Mixed precision FP16 - fp8 base - highvram - full FP16 --network_args train_blocks=single - 128 rank : 18.39 GB - 18 second / it
Accelerate launch Mixed precision BF16 - fp8 base - highvram - full BF16 --network_args train_blocks=single - 128 rank : 18.39 GB - 21 second / it
Running Experiments
1e_04_bf16_128_rank : completed - 48 gb GPU
1e_04_fp16_128_rank : completed - 48 gb GPU
1e_04_bf16_128_rank_full_bf16 : completed - 48 gb GPU
1e_04_fp16_128_rank_full_fp16 : completed - 48 gb GPU
1e_04_fp8_32_rank : completed - 23.58 GB
1e_04_fp8_full_fp16_32_rank : completed - 23.28 GB
1e_04_fp8_bf16_accelerate_32_rank : completed - 23.57 GB
1e_04_fp8_bf16_accelerate_full_bf16_32_rank : completed - 23.28 GB
Early Testing Results
Looks like FP16 accelerator training yields way inferior results compared to BF16 accelerator training
This results is very early experiments only 50 epoch
More results obtained
So far best working cases are 1e_04_bf16_128_rank_full_bf16 and 1e_04_fp8_bf16_accelerate_full_bf16_32_rank
FP16 training broken
Starting 8 more test right now
These below images has no face inpainting raw output
These are early results of my extensive FLUX LoRA training but 8 bit is way inferior to 16 bit atm
8 bit looks undertrained perhaps and 16 bit is overtrained with current LR and 150 epoch
Gonna do more testing now
No face inpainting raw 1024x1024 images
Some more full comparisons attached to attachments
New Started Trainings
48 GB Configs - uses 42.6 GB VRAM
9e_05_bf16_128_rank_full_bf16 - completed
8e_05_bf16_128_rank_full_bf16 - completed
7e_05_bf16_128_rank_full_bf16 - completed best at 150 epoch
6e_05_bf16_128_rank_full_bf16 - completed very best at 200 epoch
24 GB Configs - uses 23.30 GB VRAM
2e_04_fp8_bf16_accelerate_full_bf16_32_rank
3e_04_fp8_bf16_accelerate_full_bf16_32_rank
9e_05_fp8_bf16_accelerate_full_bf16_32_rank
8e_05_fp8_bf16_accelerate_full_bf16_32_rank
New Training Results for 48 GB Configs
So far I have tested 48 GB configs results and very best is 6e_05_bf16_128_rank_full_bf16 however it requires 200 epoch training for 15 images
At 150 epochs 7e_05_bf16_128_rank_full_bf16 yields best results but it is not as good as 200 epoch of 6e_05_bf16_128_rank_full_bf16
New grids uploaded to attachments , 50_epoch_LowerLR_48_tests.jpg , 100_epoch_LowerLR_48_tests.jpg , 150_epoch_LowerLR_48_tests.jpg , 200_epoch_LowerLR_48_tests.jpg, Face_Inpainted_48_GB_Best_Comparison.jpg
Tested configs updated to V2 - look attachments for the zip file
Here some examples
24 GB Experiments
I feel like same LR of 48 GB works
So I decided to go with single version tests from now on
At the very top of the model I added best configs
Also configs zip file updated
24 GB experiments files : 50_epoch_LowerLR_24_GB_config_tests.jpg , 100_epoch_LowerLR_24_GB_config_tests.jpg , 150_epoch_LowerLR_24_GB_config_tests.jpg , 200_epoch_LowerLR_24_GB_config_tests.jpg
Started 7 New Trainings
6e_05_best_with_reg_images - completed results below
6e_05_best_max_grad_norm_0 - completed results below
6e_05_best_raw_sigmoid - completed results below
6e_05_best_raw_sigma - completed results below
6e_05_best_additive_sigma - completed results below
6e_05_best_sigma_scaled_sigma - completed results below
6e_05_best_additive_uniform - completed results below
Prediction_Type_Timestep_Sampling_200_Epoch.jpg , Prediction_Type_Timestep_Sampling_150_Epoch.jpg
Comments
Furkan Gözükara
with FLUX yes. since we utilize RAM heavily with Block Swapping. Upgrade to 64 GB and i guarantee it will work flawlessly
Michael Liu
My RAM is 32gb. Ah i see, it’s correlated with system ram too because of my limited vram🫡
Furkan Gözükara
how much system RAM you have? probably 32 right? upgrade to 64 and it shouldnt happen anymore.
Michael Liu
Why i always run out of memory, unless enabling fp8 base ? Rtx 4060Ti 16gb vram🙏🏻
Furkan Gözükara
no FLUX works great. I had tested higher batch size with SDXL with limited test it was terrible :D you can see FLUX grids here read top to bottom please : https://www.patreon.com/posts/112099700
David Bene
But did you see any changes on face consistency? On sdxl with higher batch it looked like the face shapes for different expressions became "merged together". So like the rounded cheeks of a huge smile shown up on non smiling faces and weird things like this... Anyways i will test it out anyways because im curious, i will use your settings to make sure i nail the LR.
Furkan Gözükara
i tested batch size 1 vs 7 on flux. and batch size 7 requires new LR. you may apply square root 7 x batch size 1 LR. so if you want to apply that to SDXL you need to also amplify learning rate. moreover batch size 1 still yields the very best quality. I really cant return SDXL since its quality horrific compared to FLUX :D i plan to investigate fully SD 3.5 medium which will be lightweight and still good quality and also SD 3.5 large
David Bene
I saw you increase batch to 7 or so. With sdxl and sd1.5 it decreased the subject fidelity by a huge degree during lora training. Does it have any effect on flux finetune? Could you share test images with batch count 1 vs 7? Thanks for the great content btw.
Furkan Gözükara
please follow this video if you format pc : https://youtu.be/DrhUHnYfwC0
Jim M
Just installed 3.10 and have the same issue. Did a complete uninstall and reinstall of all apps and still quits. I'm going to try a fresh install of Win10 later tonight
Furkan Gözükara
ye please use python 3.10.11
Jim M
When doing the initial install, and selecting "1" to start the cmd windows quits. I've installed all the requirements, but using Python 3.11.
Wayne Li
Have you ever tried some specific painting style? I am struggling with it.
Furkan Gözükara
i plan to test new optimizers hopefully soon
Steve
Any thoughts on using AdEMAMix?
Furkan Gözükara
it is ok step 2 fixes xformers. also since we dont use xformers during training it is not important either
Chris Johnson
I got the following install errors: Collecting torchvision==0.19.0+cu124 Downloading https://download.pytorch.org/whl/cu124/torchvision-0.19.0%2Bcu124-cp310-cp310-win_amd64.whl (5.9 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.9/5.9 MB 25.0 MB/s eta 0:00:00 ERROR: Could not find a version that satisfies the requirement xformers==0.0.27.post2 (from versions: none) ERROR: No matching distribution found for xformers==0.0.27.post2 [notice] A new release of pip is available: 23.0.1 -> 24.2 [notice] To update, run: python.exe -m pip install --upgrade pip
Furkan Gözükara
i havent tried yet. i trained a style though it is being shared here - will update there in few hours : https://huggingface.co/MonsterMMORPG/3D-Cartoon-Style-FLUX
Robert Arsene
Have you tried training multiple subjects in the same Lora? I did 3 subjects placed in 3 different folders under the img folder. I only get black images. Nothing generated.
Furkan Gözükara
it can't be. use massed compute to train there
Dylan Normandin
Hi. so can this be used for mac? I don't have windows.
Furkan Gözükara
hello to fix this issue you need to copy paste that dll or install c++ and reinstall. your solution options shared in this post i just updated it
Furkan Gözükara
yes someone asked this to kohya. it is because of caching prompts. i think still not fixed but can you ask this here ? https://github.com/kohya-ss/sd-scripts/pull/1374
LoRAMaker4435
Hello, I'm currently using Kohya_GUI_Flux_Installer_v17.zip, which is working great BTW for training, but if I try to change the sample prompts midway though training, it hangs. Is this fixed in Kohya_GUI_Flux_Installer_v18.zip?
Moin Syed
Looks like some module is having trouble loading - Traceback (most recent call last): File "C:\Users\moinh\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\moinh\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "D:\ForgeFlux\Kohya_GUI_Flux_Installer_21\kohya_ss\venv\Scripts\accelerate.exe\__main__.py", line 4, in File "D:\ForgeFlux\Kohya_GUI_Flux_Installer_21\kohya_ss\venv\lib\site-packages\accelerate\__init__.py", line 16, in from .accelerator import Accelerator File "D:\ForgeFlux\Kohya_GUI_Flux_Installer_21\kohya_ss\venv\lib\site-packages\accelerate\accelerator.py", line 32, in import torch File "D:\ForgeFlux\Kohya_GUI_Flux_Installer_21\kohya_ss\venv\lib\site-packages\torch\__init__.py", line 148, in raise err OSError: [WinError 126] The specified module could not be found. Error loading "D:\ForgeFlux\Kohya_GUI_Flux_Installer_21\kohya_ss\venv\lib\site-packages\torch\lib\fbgemm.dll" or one of its dependencies.
Furkan Gözükara
FLUX itself tends to generate blurry backgrounds at the moment. try to add some prompts that will remove blurry backgrounds. like sharp focused i dont know try to find :D
Diggy Dre
I tried it with the same amount of Images that you used in the youtube video. For some reason, I am getting a good likeness, but the backgrounds are blurry. Did I do something wrong?
Furkan Gözükara
yes you should be able to use any flux DEV version checkpoint and it should work perfectly. but make sure that they are either fp8 or fp16 not quantized versions
meteor 1942
Thank you for the work on this one Furkan. Question: In theory, can we use any base Flux model while training the LORA so long as we use the same when it comes time later to generate images?. e.g. if we downloaded a fine-tuned FLUX checkpoint from huggingface/civitai? Or are there some limits/constraints?
Furkan Gözükara
i think it can i am still researching it and will update configs once i have perfected it hopefully
楠 陈
Clip L text encoder. After training, do you think the quality has been significantly improved?
Furkan Gözükara
video published : https://youtu.be/nySGu12Y05k
Furkan Gözükara
hello. extract the config again from zip file and directly load into lora tab. it is probably corrupted. and it shall work
Cody1611
Hello! I i stuck here: INFO no latents to cache train_util.py:1034 2024-08-28 19:52:03 INFO move vae and unet to cpu to save memory flux_train_network.py:187 Traceback (most recent call last): File "/workspace/kohya_ss/sd-scripts/flux_train_network.py", line 445, in trainer.train(args) File "/workspace/kohya_ss/sd-scripts/train_network.py", line 392, in train self.cache_text_encoder_outputs_if_needed(args, accelerator, unet, vae, text_encoders, train_dataset_group, weight_dtype) File "/workspace/kohya_ss/sd-scripts/flux_train_network.py", line 191, in cache_text_encoder_outputs_if_needed unet.to("cpu") File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1174, in to return self._apply(convert) File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 780, in _apply module._apply(fn) File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 805, in _apply param_applied = fn(param) File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1167, in convert raise NotImplementedError( NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device. Traceback (most recent call last): File "/workspace/kohya_ss/venv/bin/accelerate", line 8, in sys.exit(main()) File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main args.func(args) File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1106, in launch_command simple_launcher(args) File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['/workspace/kohya_ss/venv/bin/python', '/workspace/kohya_ss/sd-scripts/flux_train_network.py', '--config_file', '/workspace/kohya_ss/outputs/config_lora-20240828-195119.toml']' returned non-zero exit status 1. 19:52:04-971507 INFO Training has ended. Do you have any idea 48gb vram runpod flux1 is selected
Samael1976
Hi Furkan, I just write the same problem of runpod today (yesterday it worked). Yes I've select a6000 ADA and the template that you suggest in the instruction file.
Furkan Gözükara
you probably didnt load into LoRA tab. I am recording a video wait for it
Frederic Collin
Sorry to say but this is a mess ... no real conclusion, a lot of "best" everywhere but impossible to find the configs easily to use. I've tried few of them which are simply not working with my 3090.
Furkan Gözükara
it inpaint face after image generated to get better face. you can use notepad++ to replace all and remove all such part and test and see
Mike
Sorry, the most important part is probably missing. this is not transferred with the brackets. "segment:face,0.7"
Furkan Gözükara
what you mean?
Mike
Hello, why is this in the prompt?
Furkan Gözükara
awesome ty for comment. i am gonna do 8 more training today for some new feature came to kohya
Adam Chido
the new scripts worked right out of the box for me on 3 12gb machines. Thank you Furkan!
Furkan Gözükara
:D it takes time
Furkan Gözükara
t5 text encoder attention masking. it slightly improves quality
Furkan Gözükara
i dont have but i have swarmui tutorial here : https://youtu.be/bupRePUOA18
Arcon Septim
What is the difference in these two regarding quality? Rank_1_28700MB_Slow.json - 16bit - 8.53 second / it Rank_2_27360MB_Fast.json - 16bit - 4.49 second / it I see its double the time for full training....
stephan stinker
Hello everyone, can I find a workflow for confyUI anywhere?
Arcon Septim
very soon 1 week haha
Furkan Gözükara
yep caption reduces likeliness i just add grid test results for that
Furkan Gözükara
i just updated configs can you try lower VRAM configs and let me know? it doesnt tell the error reason
Franco Antonelli
can't get it to work with my RTX 3090; it runs out of memory. I followed the instructions for the update in step 2 and configured the acceleration settings as you suggested, but I'm still facing this issue. I'm not sure where I'm going wrong "You are using the default legacy behaviour of the . This is expected, and simply means that the legacy (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set legacy=False. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 INFO Using DreamBooth method. train_network.py:279 INFO prepare images. train_util.py:1803 INFO get image size from name of cache files train_util.py:1741 100%|████████████████████████████████████████████████████████████████████████████████| 19/19 [00:00<00:00, 3800.09it/s] INFO set image size from cache files: 0/19 train_util.py:1748 INFO found directory C:\koya\kohya_ss\outputs\prep\img\1_messi man contains train_util.py:1750 19 image files WARNING No caption file found for 19 images. Training will continue without train_util.py:1781 captions for these images. If class token exists, it will be used. / 19枚の画像にキャプションファイルが見つかりませんでした。これらの画像につ いてはキャプションなしで学習を続行します。class tokenが存在する場合はそれを使います。 WARNING C:\koya\kohya_ss\outputs\prep\img\1_messi man\2.PNG train_util.py:1788 WARNING C:\koya\kohya_ss\outputs\prep\img\1_messi man\Ca3ptura.PNG train_util.py:1788 WARNING C:\koya\kohya_ss\outputs\prep\img\1_messi man\Capt3ura.PNG train_util.py:1788 WARNING C:\koya\kohya_ss\outputs\prep\img\1_messi man\Capt5ura.PNG train_util.py:1788 WARNING C:\koya\kohya_ss\outputs\prep\img\1_messi man\Captura2.PNG train_util.py:1788 WARNING C:\koya\kohya_ss\outputs\prep\img\1_messi man\proxy-image (11).jpeg... train_util.py:1786 and 14 more INFO 19 train images with repeating. train_util.py:1844 INFO 0 reg images. train_util.py:1847 WARNING no regularization images / 正則化画像が見つかりませんでした train_util.py:1852 INFO [Dataset 0] config_util.py:570 batch_size: 1 resolution: (1024, 1024) enable_bucket: False network_multiplier: 1.0 [Subset 0 of Dataset 0] image_dir: "C:\koya\kohya_ss\outputs\prep\img\1_messi man" image_count: 19 num_repeats: 1 shuffle_caption: False keep_tokens: 0 keep_tokens_separator: caption_separator: , secondary_separator: None enable_wildcard: False caption_dropout_rate: 0.0 caption_dropout_every_n_epoches: 0 caption_tag_dropout_rate: 0.0 caption_prefix: None caption_suffix: None color_aug: False flip_aug: False face_crop_aug_range: None random_crop: False token_warmup_min: 1, token_warmup_step: 0, alpha_mask: False, is_reg: False class_tokens: messi man caption_extension: .txt INFO [Dataset 0] config_util.py:576 INFO loading image sizes. train_util.py:876 100%|███████████████████████████████████████████████████████████████████████████████| 19/19 [00:00<00:00, 18978.75it/s] INFO prepare dataset train_util.py:884 INFO preparing accelerator train_network.py:333 accelerator device: cuda 2024-08-26 13:37:08 INFO Building Flux model dev flux_utils.py:43 INFO Loading state dict from flux_utils.py:48 C:/Users/billi/AI/ComfyUI_windows_portable/ComfyUI/models/unet/flux1-dev.s afetensors INFO Loaded Flux: flux_utils.py:51 INFO Building CLIP flux_utils.py:70 INFO Loading state dict from flux_utils.py:163 C:/Users/billi/AI/ComfyUI_windows_portable/ComfyUI/models/clip/clip_l.saf etensors INFO Loaded CLIP: flux_utils.py:166 INFO Loading state dict from flux_utils.py:209 C:/Users/billi/AI/ComfyUI_windows_portable/ComfyUI/models/clip/t5xxl_fp16 .safetensors INFO Loaded T5xxl: flux_utils.py:212 INFO Building AutoEncoder flux_utils.py:58 INFO Loading state dict from flux_utils.py:62 C:/Users/billi/AI/ComfyUI_windows_portable/ComfyUI/models/vae/ae.sft INFO Loaded AE: flux_utils.py:65 import network module: networks.lora_flux INFO [Dataset 0] train_util.py:2326 INFO caching latents with caching strategy. train_util.py:984 INFO checking cache validity... train_util.py:994 100%|███████████████████████████████████████████████████████████████████████████████| 19/19 [00:00<00:00, 18996.85it/s] INFO caching latents... train_util.py:1038 100%|██████████████████████████████████████████████████████████████████████████████████| 19/19 [00:03<00:00, 5.16it/s] 2024-08-26 13:37:12 INFO move vae and unet to cpu to save memory flux_train_network.py:156 INFO move text encoders to gpu flux_train_network.py:164 Traceback (most recent call last): File "C:\Python3_10_11\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Python3_10_11\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\koya\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in File "C:\koya\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 48, in main args.func(args) File "C:\koya\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1106, in launch_command simple_launcher(args) File "C:\koya\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['C:\\koya\\kohya_ss\\venv\\Scripts\\python.exe', 'C:/koya/kohya_ss/sd-scripts/flux_train_network.py', '--config_file', 'C:/koya/kohya_ss/outputs/prep\\model/config_lora-20240826-133657.toml']' returned non-zero exit status 3221225477. 13:37:51-568657 INFO Training has ended"
Max
I have tried both (training realistic photos from myself) and without captioning worked better for me
Max
Use Europe if possible, haven't had any problems there.
Steve
I think the biggest issue was I was testing multi epochs using S/R script and that was causing it to have to load the new model and then move it after all generations. I switched to the regular model and it works fine for the 4070 seems like the same speed as the quantified one I was using. testing the epochs in independent generations instead of using that search replace script for each epoch vastly improved speed. sacrificing the automatic grid to see it at a glance wasnt a big deal.
Franco Antonelli
same here
Furkan Gözükara
i think people working to make work loras with such quantized dev models but not sure about latest status
Furkan Gözükara
use fp16 dev model it will cast to fp8. captioning reduces likeliness i have tested and gonna post hopefully tomorrow. for style training captions may work better though
Furkan Gözükara
yep hopefully windows, runpod and massed compute tutorials very soon
Arvin Flores
will you be doing a full flux tutorial with runpod kohya?
Steve
you were correct using the full size model to train fixed the issue. took about 25hours to train 100 epoch on a 4070 i believe total training was around 5k steps. The issue I'm running into now is that using those loras with the flux.1 dev bnb 4-bit model takes forever to generate art. ~about one hour per 35 step image.
guangyu niu
Hi, Do I need to caption the dataset? and I notice that the "use fp8 base model" is truned on, does that mean I can use fp8 dev model? and Which model give better result?
shen oracle
waiting for u
Furkan Gözükara
supposed to work in forge
ElecMat
Does the Lora is suppose to work in Forge or only in ComfyUI ?
Furkan Gözükara
no for fine tuning i will do a new research
shen oracle
你的json既可以finetune又可以lora？
Furkan Gözükara
yep runpod broken. 2.5 second great
Max
Ok, so fortunately, it does work with the A100. So probably they (runpod) have some problems with the A40 Pods in Europe, because in terms of Memory, the A100 isn't nearly at it's limits :D But it's nice to see 1014x1024 only taking around 2.5-3 seconds per iteration :D
Max
I'm right now testing on an A100 GPU to really test it out
Max
Hmmm... What images are you using? 1024x1024, or 512x512? Because i tried it on 6 different pods, and no pod seem to be working :D
Furkan Gözükara
flux.1 dev bnb 4-bit could not be supported. by the way the vram usage doesnt related to base model you pick during training. so use fp16 dev model the best handled case
Furkan Gözükara
hello just tested and it works. but A40 is slow :d https://pasteboard.co/QENsE96CfaBK.png https://pasteboard.co/jqaeX3as53C2.png probably it was broken pod
Steve
It's definitely checked. i used the flux.1 dev bnb 4-bit because I was using that for forge based on your benchmark. I figured once I got it working I would slowly work my way to higher models until I got OOM on my 12gb 4070. I haven't tried the update yet I'll be able to try it in a couple of hours.
Max
Have you found out something? I also tested a bit but still cannot find where it's hanging right now (as it worked yesterday on that machine with this setting - which is not the case anymore, tested an older version but i get the same error)
Furkan Gözükara
30 steps uni pc normal . If you have vram use dram 16bit
Arcon Septim
What are the best settings and sampler in SwarmUI to use with this flux model and lora for best generation results? Steps amount CFG etc Thank you!
Furkan Gözükara
what is 8/8? 18 gb is 1024x1024 for me :D - but it is raw. your windows will use some extra too
Furkan Gözükara
great. i use swarmui works perfect
Furkan Gözükara
i am testing right now to fix if broken thanks
Adam Chido
Now i'm just frustrated. Fresh Kohya installs on the right repo, fresh python, your fix scripts run without error, but i can't get training under like 18gb even at 512 and 8/8. I am not in a hurry. Make sure you're sleeping, lol.
Max
One moment, i have to check. edit: Yes, it's the official runpod pytorch 2.1 with python 3.10 and Cuda 11.8.0, so exactly the one you mention in your guide :)
Max
I know ;) Already found out that forge is (still after the update) trying to patch LoRas which results in huge amounts of Vram, so i switched to Comfyui and the first Lora i trained was "ok". Way to overfitted and quality is "ok" as i trained it on 512, not 1024 pixels
Furkan Gözükara
Yes it is unrelated :)
Furkan Gözükara
Did you select pytorch Cuda 11.8 template as Witten? I will test and let you know if broken or not. It is a40 gpu right?
Max
2024-08-24 13:12:04 INFO epoch is incremented. current_epoch: 0, epoch: 1 train_util.py:668 /workspace/kohya_ss/venv/lib/python3.10/site-packages/torch/utils/checkpoint.py:1399: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead. with device_autocast_ctx, torch.cpu.amp.autocast(**cpu_autocast_kwargs), recompute_context: # type: ignore[attr-defined] Traceback (most recent call last): File "/workspace/kohya_ss/sd-scripts/flux_train.py", line 905, in train(args) File "/workspace/kohya_ss/sd-scripts/flux_train.py", line 736, in train accelerator.backward(loss) File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/accelerator.py", line 2159, in backward loss.backward(**kwargs) File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/torch/_tensor.py", line 521, in backward torch.autograd.backward( File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/torch/autograd/__init__.py", line 289, in backward _engine_run_backward( File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/torch/autograd/graph.py", line 768, in _engine_run_backward return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 1116, in unpack_hook frame.recompute_fn(*args) File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 1400, in recompute_fn fn(*args, **kwargs) File "/workspace/kohya_ss/sd-scripts/library/flux_models.py", line 720, in _forward attn = attention(q, k, v, pe=pe, attn_mask=attn_mask) File "/workspace/kohya_ss/sd-scripts/library/flux_models.py", line 446, in attention x = torch.nn.functional.scaled_dot_product_attention(q, k, v, attn_mask=attn_mask) torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 972.00 MiB. GPU 0 has a total capacity of 44.35 GiB of which 541.44 MiB is free. Process 1747024 has 43.81 GiB memory in use. Of the allocated memory 41.34 GiB is allocated by PyTorch, and 2.15 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) steps: 0%| | 0/2400 [00:06 sys.exit(main()) File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main args.func(args) File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1106, in launch_command simple_launcher(args) File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['/workspace/kohya_ss/venv/bin/python', '/workspace/kohya_ss/sd-scripts/flux_train.py', '--config_file', '/workspace/kohya_ss/outputs/config_dreambooth-20240824-131040.toml']' returned non-zero exit status 1. 13:12:12-693701 INFO Training has ended. in the todays version of 48gb and even 24 GB i get this error when training on 1024*1024 images on an A40 (runpod) with 48GB of VRAM. It seems like there's an outofmemory issue because the memory handling doesn't seem to be working correctly. Do you have any ideas on how to solve this?
Furkan Gözükara
updated configs and added Windows_Download_Training_Model_Files.bat file to download necessary training model files into the bat file run directory please try newest
Furkan Gözükara
this happens when you dont select flux1 checkbox. please make sure to check it. added screenshot to the top of page. also updated configs please download latest ones and which base model are you using? i will add 1 click downloader to download all necessary base models
Steve
still with 8/24 update getting the same error: Traceback (most recent call last): File "W:\Kohya_GUI_Flux_Installer_v3\kohya_ss\sd-scripts\flux_train_network.py", line 411, in trainer.train(args) File "W:\Kohya_GUI_Flux_Installer_v3\kohya_ss\sd-scripts\train_network.py", line 342, in train model_version, text_encoder, vae, unet = self.load_target_model(args, weight_dtype, accelerator) File "W:\Kohya_GUI_Flux_Installer_v3\kohya_ss\sd-scripts\flux_train_network.py", line 65, in load_target_model model = self.prepare_split_model(model, weight_dtype, accelerator) File "W:\Kohya_GUI_Flux_Installer_v3\kohya_ss\sd-scripts\flux_train_network.py", line 98, in prepare_split_model flux_upper.to(accelerator.device, dtype=target_dtype) File "W:\Kohya_GUI_Flux_Installer_v3\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1174, in to return self._apply(convert) File "W:\Kohya_GUI_Flux_Installer_v3\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 780, in _apply module._apply(fn) File "W:\Kohya_GUI_Flux_Installer_v3\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 805, in _apply param_applied = fn(param) File "W:\Kohya_GUI_Flux_Installer_v3\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1167, in convert raise NotImplementedError( NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device. Traceback (most recent call last): File "C:\Users\SysOp\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\SysOp\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "W:\Kohya_GUI_Flux_Installer_v3\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in File "W:\Kohya_GUI_Flux_Installer_v3\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 48, in main args.func(args) File "W:\Kohya_GUI_Flux_Installer_v3\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1106, in launch_command simple_launcher(args) File "W:\Kohya_GUI_Flux_Installer_v3\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['W:\\Kohya_GUI_Flux_Installer_v3\\kohya_ss\\venv\\Scripts\\python.exe', 'W:/Kohya_GUI_Flux_Installer_v3/kohya_ss/sd-scripts/flux_train_network.py', '--config_file', 'W:/NewLoraFactory/model/config_lora-20240824-051834.toml']' returned non-zero exit status 1. 05:19:07-202926 INFO Training has ended.
Max
For me it's exactly the same, i even get an out of memory issue in forge, although i have 48GB of ram, but i assume this error is not related to lora size :)
Furkan Gözükara
we don't use reg images for FLUX
Furkan Gözükara
you are welcome. you will merge lora with base model you trained don't forget
shen oracle
感谢
Furkan Gözükara
you can merge lora and extract lora again with lower rank via kohya gui lora tool
Furkan Gözükara
hello again. please try latest Kohya_GUI_Flux_Installer_v9.zip . don't forget Fix_For_FLUX_Step_2.bat . it works on my rtx 3060 perfect but slow
shen oracle
有工具可以让我不用重新训练，而把lora文件变小吗？
Furkan Gözükara
let me test latest version gui it may have broken. gui option was not working previously that is why i had to provide that
Furkan Gözükara
with that config i am training on my home RTX 3060 you can see video here : https://www.reddit.com/r/FluxAI/comments/1ey6ie3/kohya_ss_gui_flux_lora_training_on_rtx_3060_lora/ i will hopefully make a full video very soon that should help you
Furkan Gözükara
we save as float which doubles the size. you can save as fp16. also we use 128 rank. currently training 4, 8 , 16 , 32 , 64 to compare
shen oracle
the result lora file is more the 2GB,so big?
Adam Chido
The following parameters in the 10_12_16GB_GPU configuration file are incorrect/throw errors. Consumer cards do not behave like commercial hardware: "additional_parameters": "--network_args train_blocks=single" -results in error prior to training commenceing, there is a gui option for train_blocks=single "apply_t5_attn_mask": true, - Only applies to double blocks "fp8_base": true, - fp8 is not available on consumer hardware "highvram": true, - why is highvram active on lowvram cards? Finally, these configurations, even when the appropriate corrections have been made, result in Cuda out of memory errors on 3 separate systems using 12gb cards. I do not believe this configuration is ready for release.
Pidak
you have to go to dataset preparation set instance prompt and class prompt, then set training images folder with 1 repeats and set the destination training folder, then click prepare training data and after its done in the cmd then click copy info to respective folders, it worked for me
Furkan Gözükara
great
Furkan Gözükara
you are welcome
Franco Antonelli
solved! thanks a lot
Franco Antonelli
sorry ! i can t see where the error is. is set a folder "traininng_imgs" with a subfolder with all my images. thanks
Franco Antonelli
thanks!
Furkan Gözükara
sure let me know. man is class token helps model to understand what you train. if you train woman you write woman
Max
Do we really have to write "man" at the end? Or is it just the name/triggerword? What happens if i train it on 25? I will just let it run as i'm in my 10th epoch right now (only going to 50). And i also used captioning, i will write about the results as soon as i get some :)
Furkan Gözükara
folder name is like 1_ohwx man dont do 25 make 1 and train up to 200 epoch. usually 150 is good. save checkpoints and compare them. dont use reg images they dont help leave that alone
Furkan Gözükara
for flux 1 folder we dont use reg images. they dont improve results. make repeat 1. also ohwx man as prompt works great but i didnt test captioning with flux yet. hopefully will try
Max
I just found out that even with Flux you have to name the Folders as in dreambooth. But do you have any recommendations on that? Or even a guide on what all of that means? So for me, i used 25, because i assumed it's then 25 steps per epoch, right? But what about the name and the classification? For the first lora i wanted to train it on myself, so i sticked to what you've used in the tutorial. But what if i want to train on a dog? should i then name the folder "25_[nameofdog] dog"?
Arcon Septim
what about dataset preparation tab where do I put ohwx man or woman how many repeat no regularisation folder and destination training directory??
ElecMat
I followed the 22 August 2024 Update guide but you don't mention anything about the regularization images or the .txt files whit trigger words that we need for sd 1.5 or sdxl trainings. Do we need them for Flux trainings or just 1 folder whit all the images is enough for a good Flux Lora?
Max
For me it's exactly the same. All my images are 3 Dimension (RGB) .jpg 512*512 files. And the script checks if the folder exists and contains data. However, if i click on training, i get the exact same error.
Furkan Gözükara
i am gonna write this to them now :D
Furkan Gözükara
SwarmUI grid explained here : https://youtu.be/HKX8_F1Er_w 47:13 Full guide for extremely powerful grid image generation (like X/Y/Z plot)
Andrew Tomkins
What workflow are you using to generate your results?
Arcon Septim
Massedcompute never available for training :( Runpod it seems :)
Furkan Gözükara
you didnt set your training images folder path accurately max_train_steps (0 / 1 / 1 * 200 * 1) = 0 watch this tutorial carefully to understand how kohya works until i make a video. or i give private consultation too : https://youtu.be/sBFGitIvD2A
Franco Antonelli
i got the same config as in the example but geting this error once i start training 13:55:05-144169 INFO Start training Dreambooth... 13:55:05-146170 INFO Validating lr scheduler arguments... 13:55:05-149171 INFO Validating optimizer arguments... 13:55:05-151171 INFO Validating C:/Users/billi/AI/ComfyUI_windows_portable/ComfyUI/models/loras existence and writability... SUCCESS 13:55:05-152172 INFO Validating C:/Users/billi/AI/ComfyUI_windows_portable/ComfyUI/models/unet/flux1-dev.safetensors existence... SUCCESS 13:55:05-153172 INFO Validating C:/Users/billi/AI/flux training/training_imgs existence... SUCCESS 13:55:05-157173 INFO Error: 'ok_messi' does not contain an underscore, skipping... 13:55:05-159173 INFO Regulatization factor: 1 13:55:05-160173 INFO Total steps: 0 13:55:05-163174 INFO Train batch size: 1 13:55:05-164174 INFO Gradient accumulation steps: 1 13:55:05-165174 INFO Epoch: 200 13:55:05-166175 INFO max_train_steps (0 / 1 / 1 * 200 * 1) = 0 13:55:05-167175 INFO lr_warmup_steps = 0 13:55:05-170176 INFO Saving training config to C:/Users/billi/AI/ComfyUI_windows_portable/ComfyUI/models/loras\Best_v1_5e_5_max_grad_norm_0_20240823-135505.json... 13:55:05-172176 INFO Executing command: C:\Kohya_GUI_Flux_Installer_v3\kohya_ss\venv\Scripts\accelerate.EXE launch --dynamo_backend no --dynamo_mode default --mixed_precision bf16 --num_processes 1 --num_machines 1 --num_cpu_threads_per_process 2 C:/Kohya_GUI_Flux_Installer_v3/kohya_ss/sd-scripts/flux_train.py --config_file C:/Users/billi/AI/ComfyUI_windows_portable/ComfyUI/models/loras/config_dreambooth-20240823-135505.toml 2024-08-23 13:55:15 INFO Loading settings from train_util.py:4189 C:/Users/billi/AI/ComfyUI_windows_portable/ComfyUI/models/loras/config_dreambooth-20240823-135505.toml... INFO C:/Users/billi/AI/ComfyUI_windows_portable/ComfyUI/models/loras/config_dreambooth-20240823-135505 train_util.py:42082024-08-23 13:55:15 INFO Using DreamBooth method. flux_train.py:101 WARNING ignore directory without repeats / 繰り返し回数のないディレクトリを無視します: ok_messi config_util.py:589 INFO prepare images. train_util.py:1803 INFO 0 train images with repeating. train_util.py:1844 INFO 0 reg images. train_util.py:1847 WARNING no regularization images / 正則化画像が見つかりませんでした train_util.py:1852 INFO [Dataset 0] config_util.py:570 batch_size: 1 resolution: (1024, 1024) enable_bucket: False network_multiplier: 1.0 INFO [Dataset 0] config_util.py:576 INFO loading image sizes. train_util.py:8760it [00:00, ?it/s] INFO prepare dataset train_util.py:884 ERROR No data found. Please verify the metadata file and train_data_dir option. / flux_train.py:155 画像がありません。メタデータおよびtrain_data_dirオプションを確認してください。 13:55:17-451793 INFO Training has ended.
Furkan Gözükara
thanks
George Gostyshev
Great)
Furkan Gözükara
haha i see :D i will make a step by step tutorial video
George Gostyshev
Ahh)) I mean you to do little less complex overview) Not me)
Furkan Gözükara
well if you dont publish configs publicly and link here sure
George Gostyshev
Nice research! Can I ask to make some shorter\compact version of it?
Furkan Gözükara
you are welcome thanks for support. if you write from discord it is faster
Vasiliy Bulanov
Thank you, for spending time here with dumb questions! Apreciate that!
Furkan Gözükara
nope speed is normal. but you have more than 1 repeat. make repeating 1 and it will be done in 200 * 40 * 4.8 / 60 / 60 = 11 hours at most. to further speed up you can do 150 epoch and a lower resolution like 512 but it reduces quality
Vasiliy Bulanov
Hm. Works well with 2.1 pytorch template. but...... speed is terribly low, like 434 hrs for 40 img on 200epochs: steps: 0%| | 23/320000 [01:52<434:28:02, 4.89s/it, avr_loss=0.354]
Furkan Gözükara
ye i will make video. a40 should work actually i tested on rtx 3090 and 4090 both works perfect on runpod. but i used pytorch template : RunPod Pytorch 2.1 runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04 you should use same template as me
Furkan Gözükara
probably you installed inaccurately. enter inside kohya ss and try to run gui.bat file and see if works. do you have python 3.10? i will make a video hopefully very soon. almost completed trainings
Vasiliy Bulanov
Hope your video on this topic coming soon! :) It's little bit to hard for me tackle all this minor issues :)
Vasiliy Bulanov
Yep, using LorA tab. Now having this issue with both configs "6e_05_best_raw_sigmoid.json" and latest "48_GB_GPUs.json": accelerator.unwrap_model(flux).move_to_device_except_swap_blocks(accelerator.device) # reduce peak memory usage File "/workspace/kohya_ss/sd-scripts/library/flux_models.py", line 973, in move_to_device_except_swap_blocks self.to(device) ... NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device. Using clear install with RunPod Pytorch 2.2.0 template. with A40 GPU
CodePlug
Windows_Start_Kohya_SS.bat can't open the gui..after installing all dependencies..
Furkan Gözükara
it totally depends on gpu. currently on massed compute 3000 steps takes like 4 hours - 1.2$
Furkan Gözükara
which base model you given. if you message me from discord and show screenshots i can tell error
Furkan Gözükara
please reload fresh config as LoRA not dreambooth - extract again from zip file
Furkan Gözükara
please reload fresh config as LoRA not dreambooth - extract again from zip file
Steve
Tried to use other config "10_12_16GB_GPUs" got an error with the text encoder instead: File "W:\Kohya_GUI_Flux_Installer_v3\kohya_ss\sd-scripts\flux_train_network.py", line 411, in trainer.train(args) File "W:\Kohya_GUI_Flux_Installer_v3\kohya_ss\sd-scripts\train_network.py", line 330, in train self.assert_extra_args(args, train_dataset_group) File "W:\Kohya_GUI_Flux_Installer_v3\kohya_ss\sd-scripts\flux_train_network.py", line 43, in assert_extra_args assert ( AssertionError: network for Text Encoder cannot be trained with caching Text Encoder outputs / Text Encoderの出力をキャッシュしながらText Encoderのネットワークを学習することはできません Traceback (most recent call last): File "C:\Users\SysOp\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\SysOp\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "W:\Kohya_GUI_Flux_Installer_v3\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in File "W:\Kohya_GUI_Flux_Installer_v3\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 48, in main args.func(args) File "W:\Kohya_GUI_Flux_Installer_v3\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1106, in launch_command simple_launcher(args) File "W:\Kohya_GUI_Flux_Installer_v3\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['W:\\Kohya_GUI_Flux_Installer_v3\\kohya_ss\\venv\\Scripts\\python.exe', 'W:/Kohya_GUI_Flux_Installer_v3/kohya_ss/sd-scripts/flux_train_network.py', '--config_file', 'W:/NewLoraFactory/model/config_lora-20240823-043513.toml', '--network_args', 'train_blocks=single']' returned non-zero exit status 1.
Thomas
How much time did it take to train?
Steve
(got this error trying to train I used the lowest Vram option) NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device.
Furkan Gözükara
use 24_GB_GPUs.json in newest zip file that is best
Furkan Gözükara
i never tried them :D give a try but i dont trust samples ever
Furkan Gözükara
you are welcome
Furkan Gözükara
true. i am still researching impact for big dataset and using reg images
Furkan Gözükara
ok now 27 images do this. 1 repeat and 200 epoch and save every 20 or 15 epoch and compare them because we do not use reg images. for reg images usage i am in special research
Furkan Gözükara
great
Furkan Gözükara
you dont need to reduce rank anymore. just use Fix_For_FLUX_Step_2.bat and it works as low as 17 gb with 128 rank. use latest 24gb config. but i didnt see very much difference
s h a r k e y
Sure, easy answer change repeats to 1 (1_ohwx woman for instance) that basically means the training is running through in your case the 27 images once then doing epoch 2 and so fourth and when it gets to epoch 10 it will save out a Lora file and continue on till 20 then 30 etc etc. as a rule of thumb I would recommend keeping you dataset lower than 20 (for now) this means less time per epoch but also forces you to only train the very best of your training set.
Jeroen Van Harten
Would you mind helping me out? I always used to do 20 repeats, but when I do that now with 27 images and the suggested 200 epochs I get to (540 / 1 / 1 * 200 * 1) = 108000 steps. Way more than I'm used to. Is this correct?
Jeroen Van Harten
Found it, thanks!
Giuseppe Liguori
Thanks. Sample prompts with flux still doesn't works in kohya, right?
Manpreet Singh
For 24GB GPUs, what difference do you see in generation quality and likeness between rank 16 and rank 32?
Furkan Gözükara
yes this image all you need with configs :) https://www.patreon.com/file?h=110293257&i=20234120 set as training folder as 1_ohwx man - so we do repeat 1
Manpreet Singh
A written step-by-step guide somewhere would be awesome, even without a video. Is this image of yours all I would need? https://www.patreon.com/file?h=110293257&i=20234120
Manpreet Singh
Is 1e_04_fp8_bf16_accelerate_full_bf16_32_rank.json still the best for 24GB GPUs? Was 16_rank noticeably worse?
Furkan Gözükara
yes you need to edit gui.bat file. remove update requirement part . the post has a screenshot of it look from top to bottom
Jeroen Van Harten
After I run the Fix (torch 2.4.0 + cu124 update) I get Package wrong version and it reinstalls torch 2.1.2 and cu118 when i run the gui.bat. Any ideas how to bypass that?
Furkan Gözükara
1 repeat and 200 epoch. save like every 20 epoch and compare. i have used 15 images adding training dataset screenshot to the post now so refresh
Giuseppe Liguori
Hi, How many images did you use to train your face in these test? How many repetitions and epochs?
Furkan Gözükara
use lora tab to load config - reload from 0 with my config
Furkan Gözükara
set text encoder learning rate 0.
Vasiliy Bulanov
Constantly having this issue: "AssertionError: network for Text Encoder cannot be trained with caching Text Encoder outputs". If disabled caching: "AttributeError: 'T5EncoderModel' object has no attribute 'text_model'" can't beat this thing. any tips, guys?
Furkan Gözükara
You are welcome
Furkan Gözükara
Did you Check out / switch to sd3 flux branch?
Vlad
It's amazing, thank you very much))
s h a r k e y
excellent appreciate it :D
GomezBro
I downloaded the 24 GB json, loaded Kohya and loaded the json. I changed all directories and put the fluxdev tensor in documents and pointed at it. Also put images in photos folder. I hit train button and get error: \Desktop\Kohya\kohya_ss\venv\lib\site-packages\gradio\queueing.py", line 532, in process_events response = await route_utils.call_process_api( \Desktop\Kohya\kohya_ss\venv\lib\site-packages\gradio\route_utils.py", line 276, in call_process_api output = await app.get_blocks().process_api( \Desktop\Kohya\kohya_ss\venv\lib\site-packages\gradio\blocks.py", line 1928, in process_api result = await self.call_function( \Kohya\kohya_ss\venv\lib\site-packages\gradio\blocks.py", line 1514, in call_function prediction = await anyio.to_thread.run_sync( \Desktop\Kohya\kohya_ss\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 2177, in run_sync_in_worker_thread return await future \Desktop\Kohya\kohya_ss\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 859, in run result = context.run(func, *args) \Desktop\Kohya\kohya_ss\venv\lib\site-packages\gradio\utils.py", line 832, in wrapper response = f(*args, **kwargs) \Desktop\Kohya\kohya_ss\kohya_gui\lora_gui.py", line 1171, in train_model "network_module": network_module, \Desktop\Kohya\kohya_ss\kohya_gui\lora_gui.py", line 1171, in train_model "network_module": network_module, UnboundLocalError: local variable 'network_module' referenced before assignment
Furkan Gözükara
yes just added 10gb config :)
Furkan Gözükara
i tested and it reduces likeliness a lot atm will test more
Furkan Gözükara
yes just added 10gb config :)
Furkan Gözükara
yes just added 10gb config :)
Furkan Gözükara
12 GB config eklendi
Furkan Gözükara
you are welcome sorry for delay
Furkan Gözükara
yep both coming hopefully. fine tuning next after this
Diggy Dre
Okay, awesome. Thank you so much!
Arcon Septim
Thank you for research and testing for us! I hope the video will be soon for Lora and full model fine tuning. For cloud services and PCs.
Furkan Gözükara
after LoRA Hopefully
Furkan Gözükara
for 4080 if kohya fixes the bug yes. currently sadly using 18 gb minimum which is supposed to be 12 gb. i am talking with him
楠 陈
Have you tried Dreambooth's comprehensive fine-tuning?
Furkan Gözükara
When I use reg images it is 200_ohwx man but I do 1 epoch and save based on steps like every 451 steps
Furkan Gözükara
When I don't use reg images it is 1_ohwx man
Furkan Gözükara
Henüz 12 gb çalışmıyor kohya ile konuşuyorum çalışması lazım diyor ama 18 gb kullanıyor aşırı yavaş
Cemil Hacimahmutoglu
hocam bunu 12 gb lık ekran kartı ile yaptığınız test varmı
s h a r k e y
how many repeats are you running for these tests, and how are you naming your dataset? I've had decent results naming ( 20_woman - 16 or so images - 1 epoch - then retraining the output lora about 4/5 times )
s h a r k e y
From what I've tested it wouldn't right now. I haven't seen my training take less than 18gb vram. That will change without weeks if not days though. So probably yes shortly.
Diggy Dre
Can this be done with a 4080?
Furkan Gözükara
same category harder
Furkan Gözükara
yep i will do after research is done
hazwam
can you do a video tutorial, step by step, i like video more cuz it's easier
Sugar Coat VFX Design
Which route is more difficult? I think first attempt is same category. Like Same Brand, Shoe A, Shoe B, Shoe C.
Furkan Gözükara
nice idea. are they same category or different like shoes bag and eye glasses or all shoes?
Furkan Gözükara
yes i will test its impact too once i have good base config
楠 陈
Is regularized training set currently supported?
Furkan Gözükara
for 24 gb gpu : 1e_04_fp8_bf16_accelerate_full_bf16_32_rank
Furkan Gözükara
this one but i am yet to test last 8 trainings working on them : 1e_04_bf16_128_rank_full_bf16
楠 陈
Which is the json of the best training parameters at present?
Furkan Gözükara
araştırma bitince İnşallah video yapacağım
Cemil Hacimahmutoglu
Hocam video şart :D
Sugar Coat VFX Design
For example, a Lora for a brand which has 3 products. Use one Lora to trigger their product A,B,C.
Furkan Gözükara
hopefully once i finalize settings
Manpreet Singh
Nice. When is a video coming?
Furkan Gözükara
that can be another time. but can you give me example of which concepts ? so i may make another tutorial for that
Sugar Coat VFX Design
Thanks ~! Can you teach us how to train multiple concept at the same time in the coming tutorial?
Furkan Gözükara
i couldn't find yet but researching it extensively
Vlad
Hey! Is it possible to train on 12GB VRAM ?
Furkan Gözükara
you need to git clone then do git checkout sd3-flux.1 and then install. i am preparing tutorial will show there
Josh Baker
im not clear on how to install the flux into kohya? do install kohya first then how do i add flux into it as an option?
Furkan Gözükara
You are welcome thank you for support
Yannis
You're the best. Thanks
Furkan Gözükara
just published : https://www.patreon.com/posts/forge-web-ui-and-110323512
Furkan Gözükara
sure working on it right now will publish asap
Yannis
Hey :) Can you help us installing the latest version of ForgeUi for Flux on Runpod ? :) There are no updated templates for now...
