33 Commits

Author	SHA1	Message	Date
Kohya S	2bb0f547d7	update grad hook creation to fix TE lr in sd3 fine tuning	2024-11-14 19:33:12 +09:00
Kohya S	2cb7a6db02	feat: add block swap for FLUX.1/SD3 LoRA training	2024-11-12 21:39:13 +09:00
Kohya S	5e86323f12	Update README and clean-up the code for SD3 timesteps	2024-11-07 21:27:12 +09:00
Kohya S	43849030cf	Fix to work without latent cache #1758	2024-11-06 21:33:28 +09:00
Kohya S	5e32ee26a1	fix crashing in DDP training closes #1751	2024-11-02 15:32:16 +09:00
Kohya S	82daa98fe8	remove duplicate resolution for scaled pos embed	2024-11-01 21:43:47 +09:00
Kohya S	1434d8506f	Support SD3.5M multi resolutional training	2024-10-31 19:58:22 +09:00
Kohya S	bdddc20d68	support SD3.5M	2024-10-30 12:51:49 +09:00
Kohya S	db2b4d41b9	Add dropout rate arguments for CLIP-L, CLIP-G, and T5, fix Text Encoders LoRA not trained	2024-10-27 16:42:58 +09:00
kohya-ss	d2c549d7b2	support SD3 LoRA	2024-10-25 21:58:31 +09:00
Kohya S	f8c5146d71	support block swap with fused_optimizer_pass	2024-10-24 22:02:05 +09:00
Kohya S	0286114bd2	support SD3.5L, fix final saving	2024-10-24 21:28:42 +09:00
Kohya S	623017f716	refactor SD3 CLIP to transformers etc.	2024-10-24 19:49:28 +09:00
Kohya S	2500f5a798	fix latents caching not working closes #1696	2024-10-15 07:16:34 +09:00
kohya-ss	c80c304779	Refactor caching in train scripts	2024-10-12 20:18:41 +09:00
Plat	a823fd9fb8	Improve wandb logging (#1576) ... * fix: wrong training steps were recorded to wandb, and no log was sent when logging_dir was not specified * fix: checking of whether wandb is enabled * feat: log images to wandb with their positive prompt as captions * feat: logging sample images' caption for sd3 and flux * fix: import wandb before use	2024-09-11 22:21:16 +09:00
Kohya S	92e7600cc2	Move freeze_blocks to sd3_train because it's only for sd3	2024-09-01 18:57:07 +09:00
青龍聖者@bdsqlsz	ef510b3cb9	Sd3 freeze x_block (#1417) ... * Update sd3_train.py * add freeze block lr * Update train_util.py * update	2024-09-01 18:41:01 +09:00
Kohya S	7367584e67	fix sd3 training to work without cachine TE outputs #1465	2024-08-17 14:38:34 +09:00
Kohya S	da4d0fe016	support attn mask for l+g/t5	2024-08-05 20:51:34 +09:00
Kohya S	002d75179a	sample images for training	2024-07-29 23:18:34 +09:00
Kohya S	41dee60383	Refactor caching mechanism for latents and text encoder outputs, etc.	2024-07-27 13:50:05 +09:00
Kohya S	082f13658b	reduce peak GPU memory usage before training	2024-07-12 21:28:01 +09:00
Kohya S	3d402927ef	WIP: update new latents caching	2024-07-09 23:15:38 +09:00
Kohya S	9dc7997803	fix typo	2024-07-09 20:37:00 +09:00
Kohya S	3ea4fce5e0	load models one by one	2024-07-08 22:04:43 +09:00
Kohya S	c9de7c4e9a	WIP: new latents caching	2024-07-08 19:48:28 +09:00
Kohya S	ea18d5ba6d	Fix to work full_bf16 and full_fp16.	2024-06-29 17:45:50 +09:00
Kohya S	19086465e8	Fix fp16 mixed precision, model is in bf16 without full_bf16	2024-06-29 17:21:25 +09:00
Kohya S	66cf435479	re-fix assertion ref #1389	2024-06-27 13:14:09 +09:00
Kohya S	828a581e29	fix assertion for experimental impl ref #1389	2024-06-26 20:43:31 +09:00
Kohya S	8f2ba27869	support text_encoder_batch_size for caching	2024-06-26 20:36:22 +09:00
Kohya S	d53ea22b2a	sd3 training	2024-06-23 23:38:20 +09:00