Furkan Gözükara
patreon
Full Workflow For Newbie Stable Diffusion Trainers For SD 1.5 Models & SDXL Models Training With DreamBooth & LoRA
🕑 Added 2024-02-17 01:11:42 +0000 UTCIf you are new to Stable Diffusion and want to learn easily to train it with very best possible results, this article is prepared for this purpose with everything you need.
Join discord and tell me your discord username to get a special rank : SECourses Discord
- I have developed better workflows recently than that I did with Automatic1111 DreamBooth extension. Much better than that old one. Moreover, Automatic1111 DreamBooth extension is not anymore maintained so do not use it.
- I did 120+ trainings for SDXL to find the very best workflow and settings.
- I did over 70+ trainings for SD 1.5 based models to find very best workflow and settings.
- If you have further questions please leave a comment and hopefully I will keep updating this article.
- If you are overwhelmed with all the info, I am giving 1 to 1 private lectures as well
- Message me from Discord
Requirements
- You need to have Python 3.10.11 installed. I suggest you to install as shown in below video:
- https://youtu.be/-NjNy7afOQ0
- Also install ffmpeg, C++ tools & Git.
- Don't forget to install the Visual Studio 2015, 2017, 2019, and 2022 redistributable as well.
- Install Python directly into your C drive like shown in above video.
How To Install & Use Kohya SS GUI On Windows & RunPod & Kaggle
- The big video I explained how to install and use Kohya GUI on Windows is in below video:
- https://youtu.be/sBFGitIvD2A
- The big video I explained how to install Kohya GUI on RunPod is below:
- https://youtu.be/-xEwaQ54DI4
- These 2 above videos shows how to train LoRA but everything same with DreamBooth. Just use DreamBooth tab and our shared best configs.
- Our RunPod Kohya GUI SS installer script: https://www.patreon.com/posts/84898806
- A quick video where I explained how to quickly use our Kohya GUI configs is below:
- https://youtu.be/EEV8RPohsbw
- If you can't afford RunPod and don't have a decent GPU, we have a Free Kaggle account notebook that supports Kohya SS GUI and it is shown in below video:
- https://youtu.be/16-b1AjvyBE
- Kaggle Kohya Script notebook link : https://www.patreon.com/posts/kohya-sdxl-lora-88397937
- This notebook has a cell that downloads best SDXL and SD 1.5 configs for Kaggle.
- On Kaggle, I suggest you to train SD 1.5 model. It is better since on Kaggle we can't use BF16 for SDXL training due to GPU model limitation. Kaggle GPUs are old. Also we can't train with full precision Float (FP32) due to VRAM limitation.
- However on Kaggle we can train SD 1.5 with full precision Float (FP32) therefore can obtain amazing results.
- Also on this Kaggle video (below link) I have explained my recent 200 repeat and saving every N steps checkpoints strategy which is better:
- https://youtu.be/16-b1AjvyBE
- For example if you have 16 training images, 200 repeat and 1 epoch with reg images means 16x200x1x2 = 6400 steps. The last 2 comes from reg images. So if you want to save 10 checkpoints and compare, save every 641 steps. This will give you 10 checkpoints.
- After training completed, do x/y/z checkpoint comparison and decide the best model.
- x/y/z checkpoint comparison is shown in below video:
- https://youtu.be/sBFGitIvD2A
- Why 200 repeat and 1 epoch better? Because with 200 repeating, we are able to utilize more ground truth absolute quality regularization / classification images.
Ground Truth Very Best Regularization / Classification Images
- You can see our both Man and Woman reg / class images datasets here with all resolutions you may need (all pre-prepared - takes days) : https://www.patreon.com/posts/massive-4k-woman-87700469
- These regularization / classification images are manually collected by me and prepared by me from Unsplash and sorted with best quality. The preparatın took over few weeks in total. There are 5200 images for each class Man and Woman.
After Detailer Extension
- To improve faces significantly, I use After Detailer (ADetailer) extension. This extension automatically masks the face and inpaint it with the settings you provide. This will improve face significantly. Of course, the results depends on your initial training quality and the quality and resemble and quality of the initially generated image.
- In below video how to use ADetailer extension is explained:
- https://youtu.be/16-b1AjvyBE
- Always you can look video chapters : 32:07 How to install after detailer (adetailer) extension to improve faces automatically
- After installing ADetailer extension from extensions menu, please restart the Automatic1111 for it to fully and properly install
- Specific settings for SD 1.5 and SDXL shown below in their sections
How To Crop And Resize Your Training Images
- You can use our state of the art auto crop and resize scripts. They are shared here : https://www.patreon.com/posts/sota-subject-and-88391247
- You can watch crop and resize scripts video in below link:
- https://www.youtube.com/watch?v=Fbuyu35TkE4&feature=youtu.be
- The first cropping script will zoom in the person in the image with as much as trying to match the target aspect ratio without cropping any part of the human subject. It supports may other classes as well such as : 'person', 'bicycle', 'car',... Please check the patreon post for full classes.
- Then with resize script, we focus on the face on the image and resize into exact target resolution. This crops the person body parts if necessary with focusing face.
Which Captions / Tokens To Use For Training
- For training captions / tagging, if it is a person i just use ohwx man as training token and man as reg images token. You can define them either via folder name or with image_file_name.txt files. If you are training a woman use woman or whatever you are training use that class token. Like ohwx car.
- If you want to train a style or other stuff, try ohwx + class + captioning and compare it with just ohwx class.
- If you want to learn what is ohwx and why we use such rare token watch below video:
- https://youtu.be/Bdl-jWR3Ukc
- This above is old but it is gem to learn rare tokens and class tokens
- You can use our SOTA captioning models for captioning which includes batch captioning for the following models: Clip Interrogator (115 Clip Vision Models), LLaVA (the strongest one), CogVLM, Qwen-VL, Blip2 & Kosmos-2 (even works with 2 GB VRAM GPU best one for weak GPUs).
- You can download 1 click installers and batch captioning supporting amazing Gradio web apps having scripts from here : https://www.patreon.com/posts/sota-image-for-2-90744385
- Each script has instructions and installers for Windows & RunPod. Hopefully I will add Free Kaggle notebooks as well.
- If you are local linux user, just change the paths of RunPod scripts and use them on your local machine.
- Each script has default very best image captioning prompts set in the GUI apps.
- Once you generated captions with our SOTA captioning tools, you can use Kohya GUI's amazing caption manipulation screen as shown below.
- You can download the full resolution of below image : Kohya Image Caption.png
How To Make Kohya GUI To Use Captions Instead Of Folder Names
- Once the Caption Extension is set and enabled, it will read the captions from the folder and ignore the folder names.
- The repeating count number will be still valid.
- You can download below image full size from : Kohya Enable Image Caption.png
SD 1.5 Based Models Configuration
- Download best configuration from here : https://www.patreon.com/posts/very-best-kohya-97379147
- For SD 1.5 I used 768x768 training, no bucketing (I crop training images into 768x768 aspect ratio and resize them to 768x768) and generate images in 1024x1024.
- Use same resolution of training images resolution for regularization / classification images resolution.
- Once you become a more experienced, you can enable bucketing and do bucketing enabled different aspect ratio and resolution including trainings
- I use After Detailer (ADetailer) with the following configs for SD 1.5
- Everything default except below:
- Prompt : photo of ohwx man if realism is targeted
- If you are targeting stylization such as 3D you need to try with full prompt or like 3D image of ohwx man
- Set ADetailer inpainting resolution to 768x768 : Remember we are generating images in 1024x1024 but when inpainting face, it is best to inpaint with native resolution we trained.
- The denoise strength I set is 0.5 and I also set separate number of steps for better quality like 70
- Here ADetailer settings for SD 1.5 download image to see : SD 1.5 ADetailer Settings.png
- I compared 1024x1024 training vs 768x768 training for SD 1.5 and 768x768 performed better even though we generate images in 1024x1024.
- Also not all SD 1.5 models will support 1024x1024 resolution. To find which model is best, I compared 161 SD 1.5 models. You can watch its video below:
- https://youtu.be/G-oZn4H-aHQ
- The Kohya SS GUI config for SD 1.5 automatically uses the best SD 1.5 realism model that supports 1024x1024 image generation and automatically downloads it into your PC.
SD 1.5 Based Models Results & Prompts
- 20 Images : https://civitai.com/posts/1300478
- 20 Images : https://civitai.com/posts/1300774
- 20 Images : https://civitai.com/posts/1300759
What If You Want LoRA For SD 1.5 Based Models?
- LoRA is optimized version of DreamBooth training. Therefore, no matter what configuration you use, it will be inferior to the DreamBooth training. Moreover, LoRA has infinite number of configuration combination. Therefore, I don't research LoRA.
- Also as mentioned in this paper, https://arxiv.org/abs/2402.09353, LoRA lacks a significant feature : They found that LoRA either increases or decreases magnitude and direction updates proportionally but seems to lack the capability of making only subtle directional changes as found in full fine-tuning. Therefore do full DreamBooth / Fine Tuning and extract LoRA is best.
- So what you can do If you need LoRA?
- It is so easy. We use Kohya SS GUI LoRA extraction tool. Recently I contacted Kohya for this and he even Improved its logic. Now even low VRAM GPUs can extract LoRA very well.
- Open your latest version Kohya SS GUI and extract as shown in below image
- The best model that our config auto trains on is Hyper Realism V3 which you can download safetensors file here : https://civitai.com/models/158959/hyper-realism
- You need to set your base model whatever you trained on. If you used our default config, you need to download that safetensors file and use it as base.
- You can download full resolution of below image : SD 1.5 LoRA Extract.png
SDXL Based Models Configuration
- Download best configuration from here : https://www.patreon.com/posts/89213064
- For SDXL I used 1024x1024 training, no bucketing (I crop training images into 1024x1024 aspect ratio and resize them to 1024x1024) and generate images in 1024x1024.
- Use same resolution of training images resolution for regularization / classification images resolution.
- Once you become a more experienced, you can enable bucketing and do bucketing enabled different aspect ratio and resolution including trainings
- I use After Detailer (ADetailer) with the following configs for SDXL
- Everything default except below:
- Prompt : photo of ohwx man if realism is targeted
- If you are targeting stylization such as 3D you need to try with full prompt or like 3D image of ohwx man
- The denoise strength I set is 0.5 and I also set separate number of steps for better quality like 70
- Here ADetailer settings for SDXL download image to see : SDXL ADetailer Settings.png
- For realism I still find SDXL 1.0 is best but you can try other models and compare results
- Some custom models may require longer training or higher learning rate
SDXL Based Models Results & Prompts
- 20 Images : https://civitai.com/posts/1238778
- 20 Images : https://civitai.com/posts/1238769
What If You Want LoRA For SDXL Based Models?
- LoRA is optimized version of DreamBooth training. Therefore, no matter what configuration you use, it will be inferior to the DreamBooth training. Moreover, LoRA has infinite number of configuration combination. Therefore, I don't research LoRA.
- So what you can do If you need LoRA?
- It is so easy. We use Kohya SS GUI LoRA extraction tool. Recently I contacted Kohya for this and he even Improved its logic. Now even low VRAM GPUs can extract LoRA very well.
- Open your latest version Kohya SS GUI and extract as shown in below image
- You need to set your base model whatever you trained on.
- Using LoRA on custom stylized models yields good stylized images
- You can download full resolution of below image : SDXL LoRA Extract.png
Comments
dxqbYD
I am on your discord now if it's easier to share there. Same name
dxqbYD
I still wasn't happy with some of my results, so I went back to an SD1.5 Dreambooth repository that gave me perfect results, and compared it to Kohya with my parameters, Kohya with your SD15 parameters... And confirmed it. The results of this repository are MUCH better. I am starting to think that there is something wrong with the Kohya training script. If you are interested, I can share my results with you. I have used publicly available training data. Here is the rep: https://github.com/InB4DevOps/diffusers/tree/main/examples/dreambooth Unfortunately, it doesn't support SDXL and I would like to get the same quality in SDXL
Furkan Gözükara
i tested standard lora extract it was really good
dxqbYD
I've just used extract_locon.py to convert DB to Lycoris, because that's what I have been using for SD15, and it looks great at rank 32. I remember that malcolmrey had a comparison that Lycoris has less quality loss than standard Lora, but I don't know. Might be something worth investigating. I'm gonna stop bothering you now, thank you again :)
Furkan Gözükara
true
Sarath Reddy
I am working recently on this Kohya GUI new version. I guess in the data preparation you need to give input images and pretrained model name or path is -- your base model it can hyperealism.safetensors or realisticVisionV60B1.safetensors.. etc..!! which one you want to have as a base model. Thanks.
Furkan Gözükara
I agree. I will make a new tutorial hopefully. I am waiting guy to finalize updates. he makes so much right now
Art
Hi Furkan! With the new kohya interface everything is getting more complicated... for example, I didn't figured out where to set full precision Float (FP32) for SD1.5 training, and now there are two fields whwrw to input training images (one on the Dataset preparation tab and one more next to "Pretrained model name or path"... I think this version is a bit confusionary... Can you do a screenshot to make this more clear (a screenshot with numbered steps like the one about lora extraction...)?
Furkan Gözükara
hello. sorry for late reply. if you have different aspect ratio please try these 2. first try to crop them all into 1024x1024 and train without bucketing. second train without cropping and without bucketing. third train with bucketing. and compare each other. when you have different resolutions yes you need same resolution reg images. so you can give raw images and let script to crop. or use 1536x1536 highest resolution pre-prepared images SDXL 1.0 i find still best for realism. SDXL 0.9 also good. but SDXL 1.0 loras may not work good on SDXL 0.9
Furkan Gözükara
yes it may cause over training. i had like 15 images.
So Sha
Don’t you think the model gets overtrained with 160 repeats? How many images are in your dataset in this scenario?
mike oxmaul
A lot of my images would be better off cropped at 1536*768 for example. So I should enable bucketing? Do I need mixed regularisation images? If training a person with face, is it best to just not use classification images? Is the Sdxl 1.0 base still best? I saw you do a comparison with the 0.9 which also looked good. I thought this would mean we can't use Sdxl 1.0 lora etc though.
Furkan Gözükara
it is hard to find reg images for object so you can skip it. but lets say you are training a chair and you collected chair images. only chair as caption should be sufficient
Furkan Gözükara
built in config in Kaggle loads best params for Kaggle. so use it. number of epochs in options. set it 1. and set repeating 200. i explained this logic in this kaggle video : https://youtu.be/16-b1AjvyBE
Art
Thank you Furkan... we were waiting for this one! So, to recap everything for a SD1.5 training on Kaggle, I try to make a list apart from the extensive stuff in this article: - Notebook to use: kohya-sdxl-lora-training-on-a-free-kaggle-notebook_v15.ipynb - Settings: /kaggle/working/Kaggle_SD_15.json (or do we have to use one of these: https://www.patreon.com/posts/very-best-config-97381002?) - Training Images: 768x768 n images of the subject - Reg Images: /kaggle/working/man_5200_imgs_768x768.zip OR /kaggle/working/woman_5200_imgs_768x768.zip - Training Images Repeats: 200 - No of epochs: 1 (where to set this one??? do we have to set this in the parameters tab???) - No of saving steps: [(training_images_number)x200x1x2]+1 From this list the step that confuses me is: - what json config to use? the one builth-in in kohys Kaggle notebook or one of most recent configurations files? - do we have to set manually epochs? can you explain where to set 1 epoch properly (in the previous parameters setting the epochs fiels was always "1"...)
Casper Smit
I want to train an object. Do I need reg images? And do they also need captioning?
