Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

使用AutoTS的时候message 过大有什么处理方案吗 #497

Closed
kachadx opened this issue May 27, 2024 · 1 comment
Closed

使用AutoTS的时候message 过大有什么处理方案吗 #497

kachadx opened this issue May 27, 2024 · 1 comment
Assignees

Comments

@kachadx
Copy link

kachadx commented May 27, 2024

ray.tune.error.TuneError: The Trainable/training function is too large for grpc resource limit. Check that its definition is not implicitly capturing a large array or other object in scope. Tip: use tune.with_parameters() to put large objects in the Ray object store.
Original exception: Traceback (most recent call last):
File "C:\ProgramData\anaconda3\envs\myenv\lib\site-packages\ray\tune\experiment\experiment.py", line 150, in init
self._run_identifier = Experiment.register_if_needed(run)
File "C:\ProgramData\anaconda3\envs\myenv\lib\site-packages\ray\tune\experiment\experiment.py", line 352, in register_if_needed
register_trainable(name, run_object)
File "C:\ProgramData\anaconda3\envs\myenv\lib\site-packages\ray\tune\registry.py", line 112, in register_trainable
_global_registry.register(TRAINABLE_CLASS, name, trainable)
File "C:\ProgramData\anaconda3\envs\myenv\lib\site-packages\ray\tune\registry.py", line 239, in register
self.flush_values()
File "C:\ProgramData\anaconda3\envs\myenv\lib\site-packages\ray\tune\registry.py", line 277, in flush_values
_internal_kv_put(
File "C:\ProgramData\anaconda3\envs\myenv\lib\site-packages\ray_private\client_mode_hook.py", line 103, in wrapper
return func(*args, **kwargs)
File "C:\ProgramData\anaconda3\envs\myenv\lib\site-packages\ray\experimental\internal_kv.py", line 94, in _internal_kv_put
return global_gcs_client.internal_kv_put(key, value, overwrite, namespace) == 0
File "python\ray_raylet.pyx", line 2690, in ray._raylet._auto_reconnect.wrapper
File "python\ray_raylet.pyx", line 2667, in ray._raylet._auto_reconnect.wrapper
File "python\ray_raylet.pyx", line 2804, in ray._raylet.GcsClient.internal_kv_put
File "python\ray_raylet.pyx", line 580, in ray._raylet.check_status
ray.exceptions.RpcError: Sent message larger than max (845965369 vs. 536870912)

@kachadx
Copy link
Author

kachadx commented Jun 7, 2024

使用环境变量RAY_max_grpc_message_size=1073741824

@kachadx kachadx closed this as completed Jun 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants