Furkan Gözükara
patreon
FLUX De-Distilled and Anti-Bleeding Fine-Tuning / DreamBooth & LoRA Training Experiments - Also Testing CFG Impact for Stylized Images
🕑 Added 2024-10-31 12:28:00 +0000 UTCPatreon exclusive posts index to find our scripts easily, Patreon scripts updates history to see which updates arrived to which scripts and amazing Patreon special generative scripts list that you can use in any of your task.
Join discord to get help, chat, discuss and also tell me your discord username to get your special rank : SECourses Discord
Please also Star, Watch and Fork our Stable Diffusion & Generative AI GitHub repository and join our Reddit subreddit and follow me on LinkedIn (my real profile)
=======
To understand this post, you need have fully watched this tutorial : https://youtu.be/FvpWy1x5etM
You need to have fully read this post : https://www.patreon.com/posts/112099700
When training with de-distilled models like (consolidated_s6700.safetensors or VerusVision_1.0b_Transformer_fp16.safetensors), what you need to change is from our FLUX Fine-Tuning or LoRA training configs is only single parameter which is Guidance Scale
You need to change it to 3.5 from 1
If you are training community Fine-Tuned models like PixelWave which is not De-Distilled, use and train it same way as training base FLUX Dev model
I have tested 1, 2.5 and 3.5 and 3.5 yields best performance as can be seen in attached grids below on De-Distilled model
Moreover, when doing inference, we set regular CFG to 3.5 like in Stable Diffusion models instead of setting as 1 for regular FLUX DEV models
As tested in below grids, CFG 3.5 yields best quality compared to 2.5, 3.0 for de-distilled models
Using Higher CFG on FLUX DEV for Stylized Images
Another finding of myself is that training on regular FLUX DEV model as usual but when doing inference, not using CFG Scale 1 but using higher CFG Scale when generating stylized images like anime or painting
As can be seen in grids, this is either hit or miss.
Again CFG 3.5 looks like a good value to test
Complete Test Results and Findings
I have used 28 images selected from sub-set of my 256 images to train myself
I have used Dwayne Johnson 28 images dataset fully shared here : https://www.patreon.com/posts/114972274
I have tested with the following models
FLUX DEV - base model - CFG 1
Verus Vision 1.0b - de-distilled model - CFG 3.5
PixelWave v03 - Fine-Tuned model - CFG 1
FLUX DEV-De-Distilled - CFG 3.5
You can download all above models from below repository
Conclusions
PixelWave is overfit and bad quality model for realism training
Verus Vision not working better than DEV-De-Distilled - don't see reason to use it
FLUX DEV-De-Distilled still not fixes the bleeding / mixing problem when training multiple-concept sadly
Moreover, still your training overwrites your trained subject's class, such as if you are training a man, all man becomes your trained man, you captioned or not doesn't matter since FLUX has internal auto captioning alike mechanism - image embeddings used
Or lets say I trained myself and Dwayne Johnson in the same training
My images were captioned as only ohwx
Dwayne Johnson images were captioned only as bbuk
The mixing / bleeding issue still happens at DEV-De-Distilled model
Looking at the full grids will give you exact idea
Yet DEV-De-Distilled model has lesser degree of bleed / mix problem compared to FLUX DEV model
DEV-De-Distilled is a good model as a standalone single concept training and you can see grids to decide yourself
When you train without class token, e.g. man, when you generate images sometimes it generates feminine image even if you are training a man, thus class token usage really important
For example on some images when generating with only ohwx or bbuk, it was generating woman or woman like images - the trainings were also made without class token to reduce bleeding / mixing problem
However, even though I didn't use class token, still it did bleed due to internal captioning system of FLUX and FLUX being originally distilled model
Sadly de-distilled models not reached full model level yet
Hopefully I will fully research SD 3.5 to see if it works better for multi-concept and how to train it like FLUX
Model Meanings
28_img_me_FLUX_De_Distilled
Trained only with My 28 images (subset of 256 images used in FLUX Fine Tuning / DreamBooth training) - CFG 3.5
Trained model is : https://huggingface.co/nyanko7/flux-dev-de-distill
Caption : ohwx man
28_img_me_PixelWave03
Trained only with My 28 images (subset of 256 images used in FLUX Fine Tuning / DreamBooth training) - CFG 1
Trained model is : https://civitai.com/models/141592/pixelwave?modelVersionId=992642
Caption : ohwx man
28_img_me_Raw_FLUX
Trained only with My 28 images (subset of 256 images used in FLUX Fine Tuning / DreamBooth training) - CFG 1
Trained model is : https://huggingface.co/black-forest-labs/FLUX.1-dev
Caption : ohwx man
Me_and_Rock_FLUX_De_Distilled
Trained with my 28 images and I have used Dwayne Johnson 28 images
Caption : ohwx - my images
Caption : bbuk - Dwayne Johnson images
Dataset fully shared here : https://www.patreon.com/posts/114972274
Trained model is : https://huggingface.co/nyanko7/flux-dev-de-distill
CFG 3.5
Me_and_Rock_FLUX_Raw
Trained with my 28 images and I have used Dwayne Johnson 28 images
Caption : ohwx - my images
Caption : bbuk - Dwayne Johnson images
Dataset fully shared here : https://www.patreon.com/posts/114972274
CFG 3.5
Trained model is : https://huggingface.co/black-forest-labs/FLUX.1-dev
Full Grids
De_Distilled_FLUX_Models_Dual_Concept_Dwayne_Johnson.jpg
De_Distilled_FLUX_Models_Dual_Concept_Me.jpg
De_Distilled_FLUX_Models_Single_Concept.jpg
Dual_Concept_Raw_vs_Distilled_Dwayne_Johnson_Bleed_Test.jpg
Dual_Concept_Raw_vs_Distilled_Me_Bleed_Test.jpg
Epoch_vs_CFG_Comparison_CFG3.5.jpg
FLUX_vs_PixelWave_vs_De_Distilled_vs_Verus_Vision.jpg
Inference_CFG_2.5_Training_CFG_Comparison.jpg
Inference_CFG_3.5_Training_CFG_Comparison.jpg
Inference_CFG_3_Training_CFG_Comparison.jpg
RAW_FLUX_Models_Dual_Concept_Dwayne_Johnson.jpg
RAW_FLUX_Models_Dual_Concept_Me.jpg
RAW_FLUX_Models_Single_Concept_Me.jpg
Using_Big_CFG_on_FLUX_Dev_Part1.jpg
Using_Big_CFG_on_FLUX_Dev_Part2.jpg
Comments
Furkan Gözükara
here scripts fully up to date : https://youtu.be/-uhL2nW7Ddw same zip file has dreambooth config too : 44:38 Transition to RunPod and here another runpod tutorial this is dreambooth : https://youtu.be/FvpWy1x5etM 2:01:41 RunPod i recommend watch both runpod part
Tyler
Forgive my misplaced post, but where do we find the post and tutorial on how to train a lora for SDXL or Flux on Runpod, with the .sh file to get runpod capable of the training asap?
Furkan Gözükara
it was just experimental purposes for illustration nothing political reasons. but i wont use anymore
trsh3r
wow, some of those images are very politically sad. why do you guys feel like you have to put up Trump and barfX pictures everywhere
Furkan Gözükara
for FLUX it doesnt help sadly :( i tried
Casper
Regularization imgs could help against the bleeding.
Furkan Gözükara
sadly not yet. this research was for it
alfredo
About the mixing issue (still your training overwrites your trained subject's class, such as if you are training a man, all man becomes your trained man). Do we have some way to fix it? Or some workaround?
Mario Santiago
Has this been tested yet?
Furkan Gözükara
you are welcome
Mark Sutherland
thank you, appreciate that.
Furkan Gözükara
i plan to make hopefully. i also asked comfyui and swarmui. waiting their reply
Mark Sutherland
great stuff as always, would you consider a one click installer for omnigen. It seems to be an advanced photo editor where you can swap object in the image and apparantly can run on as low as 12gb ram (cuda) cheers
Furkan Gözükara
well i think use raw base flux for single concept. it is better since during inference it will be faster.
BecauseReasons
So in a nutshell fine-tune on the de-distilled version if you're doing a single concept person.
Furkan Gözükara
this i haven't tested yet. if i can achieve better results with de-distilled can be tested. tests are still going on
Furkan Gözükara
yes i had tested it. it was worsening quality. but it improves quality on de-distilled models
Dragon DarkMoon
Hi Dr, did you try training the normal Flux.1-dev model (instead of the de-distilled one) with larger CFG value? e.g. training of CFG vector = 3.5. What's the difference between normal CFG = 1 training?
John Habermeier
Hi, this is the post I've been waiting for! Thank you. One quick question, is it possible to use the "de-distilled" model purely for inference, then extract a lora and use that with base flux1-dev? Or, will the training have the wrong impact on the base model?
