Furkan Gözükara
patreon
Training a FLUX LoRA with 256 Images Experiments - Full Workflow and Conclusions
🕑 Added 2024-09-13 15:33:00 +0000 UTCPatreon exclusive posts index to find our scripts easily, Patreon scripts updates history to see which updates arrived to which scripts and amazing Patreon special generative scripts list that you can use in any of your task.
Join discord to get help, chat, discuss and also tell me your discord username to get your special rank : SECourses Discord
Please also Star, Watch and Fork our Stable Diffusion & Generative AI GitHub repository and join our Reddit subreddit and follow me on LinkedIn (my real profile)
Details
Main FLUX training thread that you need to follow : https://www.patreon.com/posts/110879657
I have been working on training a FLUX LoRA with 256 images due to pressure of the Reddit community.
No matter how many times I told them that when you add expressions the model will generate perfectly, they don't believe :)
So I used my not ready dataset, picked 256 images to show how it can perfectly generate expressions and different perspectives.
I didn't caption them only ohwx man since captioning reduces likeliness and doesn't bring improvements.
I have done total 3 different trainings.
First test done on 8x A6000 GPU with using 4x_GPU_Rank_1_FAST_Lower_Quality config file shared here up to 200 epochs : https://www.patreon.com/posts/kohya-flux-lora-110879657
I did set the LR 2x since it was 8x GPU instead of 4x so it was 0.0002
This caused overtraining after 40 epoch and since LR was too big, it didn't learn details very well
However it even learnt my broken teeth :D https://www.reddit.com/r/StableDiffusion/comments/1fe6ar3/i_am_continuing_flux_lora_training_with_256/
The grid of first experiment is here : 256 images training 8x GPU.jfif
Then i did another training with using Rank_1_T5_XXL_39200MB_9_46_Second_IT.json on a single L40S GPU. This time I trained up to 40 epochs and this was undertrained, the LR was not big enough
You can see the grid here : 256 images training 1x GPU.jfif
As a third test I have done another training with 8x A6000 - this time I have used exactly same LR as 4x_GPU_Rank_1_SLOW_Better_Quality.json
The only difference was I have enabled T5 XXL training as well - makes minimal difference, used same LR for Clip L training
I have made the comparison as can be seen below
After doing careful analysis I decided that 80 epoch is best
80 epoch means that 80 * 256 = 20480 steps but since it was 8x GPU it was total 80 x 256 / 8 = 2560 steps
Conclusions
Instead of 4 GPUs if you use 8 GPUs do not double learning rate, keep 0.0001 - double overtrains quickly
256 images were too much since i had too many repeating clothing and especially backgrounds, so it overwritten the knowledge of the model a lot, thus overfit
So you should collect a perfect dataset that has different expressions, perspectives, distances, clothings but do not have similar or repeating ones as i did
Make sure that images sharpness, focus and lightning are as much as possibly perfect
It perfectly learnt every detail of the body and expressions that exists in the training dataset
The quality is astonishing for example below image is raw output
It can do the expressions that exists in the dataset perfectly as can be seen below
It can do distant shots very well as can be seen below all are raw images
It even learnt my broken teeth or forehead veins perfectly - learns even minor details
In the below given distant shot example the body shape pretty accurate, currently i am muscular and have some extra fat too :)
Since i had different hair lengths and styles in the training dataset, it generates mixed hairs sometimes or generate one of the style randomly
The studio shot images I got are just too amazing that no studio can take such shots that is the realism and quality
If you don't provide expression in the prompt, it sometimes generates perfectly fitting expression according to the scene and sometimes doesn't
If you generate images with big resolution like 1280x1280 or 1536x1536, you will get vertical or horizontal noticeable lines - looks bad
I shared 2x upscaled 18 example images on this Reddit post, please check all and also give an upvote I appreciate it very much : https://www.reddit.com/r/StableDiffusion/comments/1ffwvpo/tried_expressions_with_flux_lora_training_with_my/
When compared to 15 images training, the realism improved significantly (below grid)
256 images training 8x GPU lower LR and T5 Attention and T5 Training - Test 3 Grid.jfif
The head shape, body proportions, head position, perspectives, all are much better, only the environment is more overfit since I had to do more steps
Since i did more steps and got more overfit, i used Claude 3.5 (like ChatGPT) to add more details and description of backgrounds and environments to the prompts
Main FLUX training zip file updated and new prompts added https://www.patreon.com/posts/110879657
Comments
Furkan Gözükara
they are lower lora network rank - dimension. you can also lower lora rank, but it will reduce quality i tested
Sonivas Sx
why when i train LoRa models are all around 1.5Gb big but when downloading Loras from civitai are 15-70Mb ? How can i reduce the size but keep the quality as high as possible?
Furkan Gözükara
ye please do ty
Voplica
We will try training this week on higher res aspect bucketing (up to 2 MP). I assume it should help, but I will post an update later if there are any meaningful results.
Furkan Gözükara
i have trained higher res and it had some improvement but not much. bucketing higher res might help but i havent tested
Voplica
Looks great! I wonder if the FLUX dev model, being a 2 MP model, might learn even better if it were trained on higher resolution images like 1440x1440. Additionally, aspect ratio bucketing with different resolutions such as 1024x1024, 1440x1440 and 864x1536 could help mitigate some of the issues encountered when generating higher-resolution images.
Furkan Gözükara
i think captioning makes minimal impact due to internal captioning, people tells that T5 impact is more important when you train text having images. i havent tested this yet
Patrick Major
I've done a tonne of testing using your rank 1 4x a6000 kohya settings with and without captions and have come to the same conclusion. Though I also like to train locally using the rank 8 or 9 depending on what I'm doing. In the higher rank files you have, the t5 encoder training is off. Would this then mean the captioning is way more important, or do you think the internal captioning still works? I haven't had time yet to train in that scenario and was wondering if you had... it would give to give guidance in the sense that if you want to skip the captioning use the lower rank training files with the t5 training on as the system auto captions and the higher rank ones need captions more?
Furkan Gözükara
i suggest this. first crop and have a baseline model. then you can try without cropping and enable bucketing and then compare. 1024 works great though
Furkan Gözükara
i made this comparison with joy caption which really fully captions the image. the thing is that i didnt see any improvements in generalization. also even if i dont caption, since flux has internal captioning alike system, all my images are still getting effect of fully captioned
cool1
You said "I didn't caption them only ohwx man since captioning reduces likeliness and doesn't bring improvements.". If there was a pre-caption option and you entered "ohwx man" (or other specific name of the person) as well as having an auto-caption/manually editing them later, wouldn't that allow good likeness as well as allowing it to better know the full scene info? If the auto captions just say describe "a man.." and the describe look and scene then it won't be as specific as entering the precaption/manually entering the name of the person.
ZXM
Great work, thanks a lot. I got one quick question here, is it necessary to crop the data sets to 1024x1024? I think previously for SDXL, it seems no need to crop and kohya ss could support various resolution with the bucket option.
Furkan Gözükara
no we have configs as low as 8gb. 24 gb cards can train almost maximum quality use rank 3 - if it be too slow use rank 4 : https://www.patreon.com/posts/110879657
Robert Morton
Is this only possible with high amounts of VRAM? Current restricted to 24GB for me
Furkan Gözükara
thanks for me as well
Sébastien Auger
Very nice new training! For me it’s the best you did 👍🏻 Reddit was right to force you to create another dataset wirh the critics, you proves that your parameters are very good, and that your work is of quality and that you listen to the community, bravo
Furkan Gözükara
yes i will share my experience after concluded. i may make a full fine tuning of flux i have such idea with 20k images
Arcon Septim
So are you going to tell us afterwards what would be the ideal steps count, repetitions etc. for having 200 photos for training source? Also are you planning on creating a full flux checkpoint fine tune for certain model? Thanks
Furkan Gözükara
Yep :)
Furkan Gözükara
Thank you so much
Furkan Gözükara
May help I haven't tested yet. Currently all are auto cropped to 1024*1024
Samuel
This is another story. ^^
Robert Arsene
What about training with different resolution like 720x1280? Would that help with full body generations at that resolution?
Patrick Major
Really great work! I had done stuff like this for SD1.5 and XL LoRA training, took forever to figure out good settings. Your work is quite appreciated.
Furkan Gözükara
thank you for your support as well
Clinton
That is a lot of work! Thank you for all you do, sir.
