[ERROR] Command exited with code 1 [2024-09-14 13:44:41] [INFO] Runner: <LogsViewRunner nb_logs=127 exit_code=1>,What should I do?

[zxtmzxtm]

[2024-09-14 13:44:05] [INFO] Running f:\Users\Administrator\pinokio\api\fluxgym.git\train.bat
[2024-09-14 13:44:05] [INFO]
[2024-09-14 13:44:05] [INFO] (env) (base) f:\Users\Administrator\pinokio\api\fluxgym.git>accelerate launch --mixed_precision bf16 --num_cpu_threads_per_process 1 sd-scripts/flux_train_network.py --pretrained_model_name_or_path "f:\Users\Administrator\pinokio\api\fluxgym.git\models\unet\flux1-dev.sft" --clip_l "f:\Users\Administrator\pinokio\api\fluxgym.git\models\clip\clip_l.safetensors" --t5xxl "f:\Users\Administrator\pinokio\api\fluxgym.git\models\clip\t5xxl_fp16.safetensors" --ae "f:\Users\Administrator\pinokio\api\fluxgym.git\models\vae\ae.sft" --cache_latents_to_disk --save_model_as safetensors --sdpa --persistent_data_loader_workers --max_data_loader_n_workers 2 --seed 42 --gradient_checkpointing --mixed_precision bf16 --save_precision bf16 --network_module networks.lora_flux --network_dim 4 --optimizer_type adafactor --optimizer_args "relative_step=False" "scale_parameter=False" "warmup_init=False" --split_mode --network_args "train_blocks=single" --lr_scheduler constant_with_warmup --max_grad_norm 0.0 --learning_rate 8e-4 --cache_text_encoder_outputs --cache_text_encoder_outputs_to_disk --fp8_base --highvram --max_train_epochs 16 --save_every_n_epochs 4 --dataset_config "f:\Users\Administrator\pinokio\api\fluxgym.git\dataset.toml" --output_dir "f:\Users\Administrator\pinokio\api\fluxgym.git\outputs" --output_name pp --timestep_sampling shift --discrete_flow_shift 3.1582 --model_prediction_type raw --guidance_scale 1 --loss_type l2
[2024-09-14 13:44:21] [INFO] The following values were not passed to accelerate launch and had defaults used instead:
[2024-09-14 13:44:21] [INFO] --num_processes was set to a value of 1
[2024-09-14 13:44:21] [INFO] --num_machines was set to a value of 1
[2024-09-14 13:44:21] [INFO] --dynamo_backend was set to a value of 'no'
[2024-09-14 13:44:21] [INFO] To avoid this warning pass in values for each of the problematic parameters or run accelerate config.
[2024-09-14 13:44:34] [INFO] f:\Users\Administrator\pinokio\api\fluxgym.git\env\lib\site-packages\diffusers\utils\outputs.py:63: FutureWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
[2024-09-14 13:44:34] [INFO] torch.utils._pytree._register_pytree_node(
[2024-09-14 13:44:35] [INFO] f:\Users\Administrator\pinokio\api\fluxgym.git\env\lib\site-packages\diffusers\utils\outputs.py:63: FutureWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
[2024-09-14 13:44:35] [INFO] torch.utils._pytree._register_pytree_node(
[2024-09-14 13:44:36] [INFO] highvram is enabled / highvramが有効です
[2024-09-14 13:44:36] [INFO] 2024-09-14 13:44:36 WARNING cache_latents_to_disk is train_util.py:3936
[2024-09-14 13:44:36] [INFO] enabled, so cache_latents is
[2024-09-14 13:44:36] [INFO] also enabled /
[2024-09-14 13:44:36] [INFO] cache_latents_to_diskが有効なた
[2024-09-14 13:44:36] [INFO] め、cache_latentsを有効にします
[2024-09-14 13:44:36] [INFO] 2024-09-14 13:44:36 INFO t5xxl_max_token_length: flux_train_network.py:155
[2024-09-14 13:44:36] [INFO] 512
[2024-09-14 13:44:37] [INFO] f:\Users\Administrator\pinokio\api\fluxgym.git\env\lib\site-packages\transformers\tokenization_utils_base.py:1601: FutureWarning: clean_up_tokenization_spaces was not set. It will be set to True by default. This behavior will be depracted in transformers v4.45, and will be then set to False by default. For more details check this issue: huggingface/transformers#31884
[2024-09-14 13:44:37] [INFO] warnings.warn(
[2024-09-14 13:44:37] [INFO] You are using the default legacy behaviour of the <class 'transformers.models.t5.tokenization_t5.T5Tokenizer'>. This is expected, and simply means that the legacy (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set legacy=False. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in huggingface/transformers#24565
[2024-09-14 13:44:38] [INFO] 2024-09-14 13:44:38 INFO Loading dataset config from train_network.py:280
[2024-09-14 13:44:38] [INFO] f:\Users\Administrator\pinoki
[2024-09-14 13:44:38] [INFO] o\api\fluxgym.git\dataset.tom
[2024-09-14 13:44:38] [INFO] l
[2024-09-14 13:44:38] [INFO] INFO prepare images. train_util.py:1807
[2024-09-14 13:44:38] [INFO] INFO get image size from name of train_util.py:1745
[2024-09-14 13:44:38] [INFO] cache files
[2024-09-14 13:44:38] [INFO] 0%| | 0/4 [00:00<?, ?it/s]
100%|██████████| 4/4 [00:00<00:00, 1998.72it/s]
[2024-09-14 13:44:38] [INFO] INFO set image size from cache train_util.py:1752
[2024-09-14 13:44:38] [INFO] files: 0/4
[2024-09-14 13:44:38] [INFO] INFO found directory train_util.py:1754
[2024-09-14 13:44:38] [INFO] f:\Users\Administrator\pinokio
[2024-09-14 13:44:38] [INFO] api\fluxgym.git\datasets\pp
[2024-09-14 13:44:38] [INFO] contains 4 image files
[2024-09-14 13:44:38] [INFO] INFO 40 train images with repeating. train_util.py:1848
[2024-09-14 13:44:38] [INFO] INFO 0 reg images. train_util.py:1851
[2024-09-14 13:44:38] [INFO] WARNING no regularization images / train_util.py:1856
[2024-09-14 13:44:38] [INFO] 正則化画像が見つかりませんでし
[2024-09-14 13:44:38] [INFO] た
[2024-09-14 13:44:38] [INFO] INFO [Dataset 0] config_util.py:570
[2024-09-14 13:44:38] [INFO] batch_size: 1
[2024-09-14 13:44:38] [INFO] resolution: (512, 512)
[2024-09-14 13:44:38] [INFO] enable_bucket: False
[2024-09-14 13:44:38] [INFO] network_multiplier: 1.0
[2024-09-14 13:44:38] [INFO]
[2024-09-14 13:44:38] [INFO] [Subset 0 of Dataset 0]
[2024-09-14 13:44:38] [INFO] image_dir:
[2024-09-14 13:44:38] [INFO] "f:\Users\Administrator\pinokio
[2024-09-14 13:44:38] [INFO] \api\fluxgym.git\datasets\pp"
[2024-09-14 13:44:38] [INFO] image_count: 4
[2024-09-14 13:44:38] [INFO] num_repeats: 10
[2024-09-14 13:44:38] [INFO] shuffle_caption: False
[2024-09-14 13:44:38] [INFO] keep_tokens: 1
[2024-09-14 13:44:38] [INFO] keep_tokens_separator:
[2024-09-14 13:44:38] [INFO] caption_separator: ,
[2024-09-14 13:44:38] [INFO] secondary_separator: None
[2024-09-14 13:44:38] [INFO] enable_wildcard: False
[2024-09-14 13:44:38] [INFO] caption_dropout_rate: 0.0
[2024-09-14 13:44:38] [INFO] caption_dropout_every_n_epo
[2024-09-14 13:44:38] [INFO] ches: 0
[2024-09-14 13:44:38] [INFO] caption_tag_dropout_rate:
[2024-09-14 13:44:38] [INFO] 0.0
[2024-09-14 13:44:38] [INFO] caption_prefix: None
[2024-09-14 13:44:38] [INFO] caption_suffix: None
[2024-09-14 13:44:38] [INFO] color_aug: False
[2024-09-14 13:44:38] [INFO] flip_aug: False
[2024-09-14 13:44:38] [INFO] face_crop_aug_range: None
[2024-09-14 13:44:38] [INFO] random_crop: False
[2024-09-14 13:44:38] [INFO] token_warmup_min: 1,
[2024-09-14 13:44:38] [INFO] token_warmup_step: 0,
[2024-09-14 13:44:38] [INFO] alpha_mask: False,
[2024-09-14 13:44:38] [INFO] is_reg: False
[2024-09-14 13:44:38] [INFO] class_tokens: txzm
[2024-09-14 13:44:38] [INFO] caption_extension: .txt
[2024-09-14 13:44:38] [INFO]
[2024-09-14 13:44:38] [INFO]
[2024-09-14 13:44:38] [INFO] INFO [Dataset 0] config_util.py:576
[2024-09-14 13:44:38] [INFO] INFO loading image sizes. train_util.py:880
[2024-09-14 13:44:38] [INFO] 0%| | 0/4 [00:00<?, ?it/s]
100%|██████████| 4/4 [00:00<00:00, 4039.78it/s]
[2024-09-14 13:44:38] [INFO] INFO prepare dataset train_util.py:888
[2024-09-14 13:44:38] [INFO] INFO preparing accelerator train_network.py:345
[2024-09-14 13:44:38] [INFO] accelerator device: cuda
[2024-09-14 13:44:38] [INFO] INFO Building Flux model dev flux_utils.py:45
[2024-09-14 13:44:39] [INFO] 2024-09-14 13:44:39 INFO Loading state dict from flux_utils.py:52
[2024-09-14 13:44:39] [INFO] f:\Users\Administrator\pinokio\ap
[2024-09-14 13:44:39] [INFO] i\fluxgym.git\models\unet\flux1-d
[2024-09-14 13:44:39] [INFO] ev.sft
[2024-09-14 13:44:39] [INFO] Traceback (most recent call last):
[2024-09-14 13:44:39] [INFO] File "f:\Users\Administrator\pinokio\api\fluxgym.git\sd-scripts\library\flux_utils.py", line 37, in load_safetensors
[2024-09-14 13:44:39] [INFO] return load_file(path, device=device)
[2024-09-14 13:44:39] [INFO] File "f:\Users\Administrator\pinokio\api\fluxgym.git\env\lib\site-packages\safetensors\torch.py", line 313, in load_file
[2024-09-14 13:44:39] [INFO] with safe_open(filename, framework="pt", device=device) as f:
[2024-09-14 13:44:39] [INFO] safetensors_rust.SafetensorError: Error while deserializing header: MetadataIncompleteBuffer
[2024-09-14 13:44:39] [INFO]
[2024-09-14 13:44:39] [INFO] During handling of the above exception, another exception occurred:
[2024-09-14 13:44:39] [INFO]
[2024-09-14 13:44:39] [INFO] Traceback (most recent call last):
[2024-09-14 13:44:39] [INFO] File "f:\Users\Administrator\pinokio\api\fluxgym.git\sd-scripts\flux_train_network.py", line 519, in
[2024-09-14 13:44:39] [INFO] trainer.train(args)
[2024-09-14 13:44:39] [INFO] File "f:\Users\Administrator\pinokio\api\fluxgym.git\sd-scripts\train_network.py", line 354, in train
[2024-09-14 13:44:39] [INFO] model_version, text_encoder, vae, unet = self.load_target_model(args, weight_dtype, accelerator)
[2024-09-14 13:44:39] [INFO] File "f:\Users\Administrator\pinokio\api\fluxgym.git\sd-scripts\flux_train_network.py", line 71, in load_target_model
[2024-09-14 13:44:39] [INFO] model = flux_utils.load_flow_model(
[2024-09-14 13:44:39] [INFO] File "f:\Users\Administrator\pinokio\api\fluxgym.git\sd-scripts\library\flux_utils.py", line 53, in load_flow_model
[2024-09-14 13:44:39] [INFO] sd = load_safetensors(ckpt_path, device=str(device), disable_mmap=disable_mmap, dtype=dtype)
[2024-09-14 13:44:39] [INFO] File "f:\Users\Administrator\pinokio\api\fluxgym.git\sd-scripts\library\flux_utils.py", line 39, in load_safetensors
[2024-09-14 13:44:39] [INFO] return load_file(path) # prevent device invalid Error
[2024-09-14 13:44:39] [INFO] File "f:\Users\Administrator\pinokio\api\fluxgym.git\env\lib\site-packages\safetensors\torch.py", line 313, in load_file
[2024-09-14 13:44:39] [INFO] with safe_open(filename, framework="pt", device=device) as f:
[2024-09-14 13:44:39] [INFO] safetensors_rust.SafetensorError: Error while deserializing header: MetadataIncompleteBuffer
[2024-09-14 13:44:40] [INFO] Traceback (most recent call last):
[2024-09-14 13:44:40] [INFO] File "f:\Users\Administrator\pinokio\bin\miniconda\lib\runpy.py", line 196, in _run_module_as_main
[2024-09-14 13:44:40] [INFO] return _run_code(code, main_globals, None,
[2024-09-14 13:44:40] [INFO] File "f:\Users\Administrator\pinokio\bin\miniconda\lib\runpy.py", line 86, in run_code
[2024-09-14 13:44:40] [INFO] exec(code, run_globals)
[2024-09-14 13:44:40] [INFO] File "f:\Users\Administrator\pinokio\api\fluxgym.git\env\Scripts\accelerate.exe_main.py", line 7, in
[2024-09-14 13:44:40] [INFO] File "f:\Users\Administrator\pinokio\api\fluxgym.git\env\lib\site-packages\accelerate\commands\accelerate_cli.py", line 48, in main
[2024-09-14 13:44:40] [INFO] args.func(args)
[2024-09-14 13:44:40] [INFO] File "f:\Users\Administrator\pinokio\api\fluxgym.git\env\lib\site-packages\accelerate\commands\launch.py", line 1106, in launch_command
[2024-09-14 13:44:40] [INFO] simple_launcher(args)
[2024-09-14 13:44:40] [INFO] File "f:\Users\Administrator\pinokio\api\fluxgym.git\env\lib\site-packages\accelerate\commands\launch.py", line 704, in simple_launcher
[2024-09-14 13:44:40] [INFO] raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
[2024-09-14 13:44:40] [INFO] subprocess.CalledProcessError: Command '['f:\Users\Administrator\pinokio\api\fluxgym.git\env\Scripts\python.exe', 'sd-scripts/flux_train_network.py', '--pretrained_model_name_or_path', 'f:\Users\Administrator\pinokio\api\fluxgym.git\models\unet\flux1-dev.sft', '--clip_l', 'f:\Users\Administrator\pinokio\api\fluxgym.git\models\clip\clip_l.safetensors', '--t5xxl', 'f:\Users\Administrator\pinokio\api\fluxgym.git\models\clip\t5xxl_fp16.safetensors', '--ae', 'f:\Users\Administrator\pinokio\api\fluxgym.git\models\vae\ae.sft', '--cache_latents_to_disk', '--save_model_as', 'safetensors', '--sdpa', '--persistent_data_loader_workers', '--max_data_loader_n_workers', '2', '--seed', '42', '--gradient_checkpointing', '--mixed_precision', 'bf16', '--save_precision', 'bf16', '--network_module', 'networks.lora_flux', '--network_dim', '4', '--optimizer_type', 'adafactor', '--optimizer_args', 'relative_step=False', 'scale_parameter=False', 'warmup_init=False', '--split_mode', '--network_args', 'train_blocks=single', '--lr_scheduler', 'constant_with_warmup', '--max_grad_norm', '0.0', '--learning_rate', '8e-4', '--cache_text_encoder_outputs', '--cache_text_encoder_outputs_to_disk', '--fp8_base', '--highvram', '--max_train_epochs', '16', '--save_every_n_epochs', '4', '--dataset_config', 'f:\Users\Administrator\pinokio\api\fluxgym.git\dataset.toml', '--output_dir', 'f:\Users\Administrator\pinokio\api\fluxgym.git\outputs', '--output_name', 'pp', '--timestep_sampling', 'shift', '--discrete_flow_shift', '3.1582', '--model_prediction_type', 'raw', '--guidance_scale', '1', '--loss_type', 'l2']' returned non-zero exit status 1.
[2024-09-14 13:44:41] [ERROR] Command exited with code 1
[2024-09-14 13:44:41] [INFO] Runner:

[Bas667]

i don't have a solution, but i get the same error

[steffenthomas]

The same error here. :-(
I am using Fluxgym into Pinokio on a Windows 11 machine.

[Bas667]

I also tried the manual installation - same error
(RTX 4060ti 16GB, Windows 11)

[Bas667]

I have found a solution, maybe it will help you:
I simply downloaded the model again. Now everything seems to work. (probably something went wrong with my first download...)

[steffenthomas]

@Bas667
Did you Download all the safetensor files and place them manually in the right folders?

[Bas667]

@steffenthomas
yes, in the log there is this line:

[2024-09-14 13:44:39] [INFO] safetensors_rust.SafetensorError: Error while deserializing header: MetadataIncompleteBuffer

so i googled for it and found out that it could be due to a corrupt safetensor file. I downloaded all the models manually again (via the link in this repro) and put them in the appropriate folders. Could already create my first loras :)

[steffenthomas]

@Bas667
At first I thought I was doing something wrong because I had used the corresponding models from my ComfyUI installation and it didn't work. However, the models there had the extension *.safetensors instead of *.sft. I have now set up a symlink that points to the ComfyUI directories so that I don't have to keep the huge models twice and now everything works! Many thanks!

[BrokenEnigma]

I'm having the same error, tried downloading the files again for the Vae etc, still get the error.

Not sure what else to try, can get log later if needed.

[maskontheface]

Has anyone managed to fix this error? I reinstalled the program several times, nothing helped

[steffenthomas]

@maskontheface
I had compared the models (VAE, CLIP and UNET) with the ones I had previously used for ComfyUI and found that the model for the UNET was different. So download this UNET model for your FLuxGym and it should work! Here is the link: https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/flux1-dev.safetensors

[BrokenEnigma]

Managed to solve mine by making sure all the required files were correct, updating everything to the latest version and doing a reboot afterwards.

Seems ok now....

[Steinix75]

I simply used short image captions and it worked well finally. Eventually you need to remove special characters, too.