Training was successful on a single card 4090GPU, but an error was reported on a 3*4090GPU. why

[orderer0001]

(lmflow_train) root@duxact:/data/projects/lmflow/LMFlow# ./scripts/run_finetune_with_lisa.sh
--model_name_or_path /data/guihunmodel8.8B
--dataset_path /data/projects/lmflow/case_report_data
--output_model_path /data/projects/lmflow/guihun_fintune_model
--lisa_activated_layers 1
--lisa_interval_steps 20
[2024-05-22 14:32:20,602] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/root/anaconda3/envs/lmflow_train/lib/python3.9/site-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
warnings.warn(
Traceback (most recent call last):
File "/data/projects/lmflow/LMFlow/examples/finetune.py", line 61, in
main()
File "/data/projects/lmflow/LMFlow/examples/finetune.py", line 44, in main
model_args, data_args, pipeline_args = parser.parse_args_into_dataclasses()
File "/root/anaconda3/envs/lmflow_train/lib/python3.9/site-packages/transformers/hf_argparser.py", line 339, in parse_args_into_dataclasses
obj = dtype(**inputs)
File "", line 135, in init
File "/root/anaconda3/envs/lmflow_train/lib/python3.9/site-packages/transformers/training_args.py", line 1641, in post_init
and (self.device.type == "cpu" and not is_torch_greater_or_equal_than_2_3)
File "/root/anaconda3/envs/lmflow_train/lib/python3.9/site-packages/transformers/training_args.py", line 2149, in device
return self._setup_devices
File "/root/anaconda3/envs/lmflow_train/lib/python3.9/site-packages/transformers/utils/generic.py", line 59, in get
cached = self.fget(obj)
File "/root/anaconda3/envs/lmflow_train/lib/python3.9/site-packages/transformers/training_args.py", line 2081, in _setup_devices
self.distributed_state = PartialState(
File "/root/anaconda3/envs/lmflow_train/lib/python3.9/site-packages/accelerate/state.py", line 293, in init
raise NotImplementedError(
NotImplementedError: Using RTX 4000 series doesn't support faster communication broadband via P2P or IB. Please set NCCL_P2P_DISABLE="1" and NCCL_IB_DISABLE="1" or use accelerate launch` which will do this automatically.

[wheresmyhair]

Thanks for your interest in LMFlow! Currently we are working on the full multi-GPU support for LISA. Please stay tuned for our latest update, thanks for your understanding 🙏