Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem in parsing waymo dataset #42

Open
KaiLong1 opened this issue Dec 8, 2023 · 7 comments
Open

Problem in parsing waymo dataset #42

KaiLong1 opened this issue Dec 8, 2023 · 7 comments

Comments

@KaiLong1
Copy link

KaiLong1 commented Dec 8, 2023

I attempted to download the Waymo dataset from the official website, but encountered the following bug while parsing the data.

Traceback (most recent call last):
File "/home/longk/anaconda3/envs/detzero/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/longk/anaconda3/envs/detzero/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/mnt/disk8Tone/longkai/DetZero/detection/detzero_det/datasets/waymo/waymo_preprocess.py", line 241, in
create_waymo_infos(
File "/mnt/disk8Tone/longkai/DetZero/detection/detzero_det/datasets/waymo/waymo_preprocess.py", line 70, in create_waymo_infos
waymo_infos_train = get_infos_worker(
File "/mnt/disk8Tone/longkai/DetZero/detection/detzero_det/datasets/waymo/waymo_preprocess.py", line 36, in get_infos_worker
sequence_infos = list(tqdm(executor.map(process_single_sequence, sample_sequence_file_list),
File "/home/longk/anaconda3/envs/detzero/lib/python3.8/site-packages/tqdm/std.py", line 1182, in iter
for obj in iterable:
File "/home/longk/anaconda3/envs/detzero/lib/python3.8/concurrent/futures/_base.py", line 619, in result_iterator
yield fs.pop().result()
File "/home/longk/anaconda3/envs/detzero/lib/python3.8/concurrent/futures/_base.py", line 444, in result
return self.__get_result()
File "/home/longk/anaconda3/envs/detzero/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/home/longk/anaconda3/envs/detzero/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/mnt/disk8Tone/longkai/DetZero/detection/detzero_det/datasets/waymo/waymo_utils.py", line 206, in process_single_sequence_and_save
process_single_tfrecord_multiprocessing(
File "/mnt/disk8Tone/longkai/DetZero/detection/detzero_det/datasets/waymo/waymo_utils.py", line 250, in process_single_tfrecord_multiprocessing
for cnt, data in enumerate(dataset):
File "/home/longk/anaconda3/envs/detzero/lib/python3.8/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 761, in next
return self._next_internal()
File "/home/longk/anaconda3/envs/detzero/lib/python3.8/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 744, in _next_internal
ret = gen_dataset_ops.iterator_get_next(
File "/home/longk/anaconda3/envs/detzero/lib/python3.8/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 2728, in iterator_get_next
_ops.raise_from_not_ok_status(e, name)
File "/home/longk/anaconda3/envs/detzero/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 6941, in raise_from_not_ok_status
six.raise_from(core._status_to_exception(e.code, message), None)
File "", line 3, in raise_from
tensorflow.python.framework.errors_impl.NotFoundError: /mnt/disk8Tone/longkai/DetZero/data/waymo/raw_data/segment-1005081002024129653_5313_150_5333_150_with_camera_labels_with_camera_labels.tfrecord; No such file or directory [Op:IteratorGetNext]

I removed '_with_camera_labels.tfrecord' from the line sequence_file_tfrecord = sequence_file[:-9] + '_with_camera_labels.tfrecord'

@KaiLong1 KaiLong1 changed the title How to download the waymo dataset How to download the waymo data Dec 8, 2023
@KaiLong1 KaiLong1 changed the title How to download the waymo data Problem in parsing waymo dataset Dec 8, 2023
@YuRui-Learning
Copy link

hello,i also meet this problem,and fix by index modify in /root/DetZero/detection/detzero_det/datasets/waymo/waymo_utils.py
sequence_file_tfrecord = sequence_file[:-28] + '_with_camera_labels.tfrecord'
but i find prepare data is chucked ,do u meet this problem?like
0%| | 0/24 [00:00<?, ?it/s]

@KaiLong1
Copy link
Author

I can parse it here, but there is an issue with the parsed data. Visualization reveals that the point cloud is messy.

And the parsing speed is very slow, far slower than the parsing data of OpenPCDET.
微信图片_20231211213826

微信图片_20231211213817

@KaiLong1
Copy link
Author

I have reanalyzed some waymo data, but it is still this error. But I have no problem parsing with OpenPCDET. I haven't made any changes to the code, why did this error occur?

@L00qqq
Copy link

L00qqq commented Feb 1, 2024

你好,我也遇到这个问题,通过修改 /root/DetZero/detection/detzero_det/datasets/waymo/waymo_utils.py 中的索引来修复, sequence_file_tfrecord = sequence_file[:-28] + '_with_camera_labels.tfrecord' 但是我发现准备数据被卡住了,你遇到这个问题吗?比如 0%| | 0/24 [00:00<?, ?it/s]

hello,i also meet this problem,and fix by index modify in /root/DetZero/detection/detzero_det/datasets/waymo/waymo_utils.py sequence_file_tfrecord = sequence_file[:-28] + '_with_camera_labels.tfrecord' but i find prepare data is chucked ,do u meet this problem?like 0%| | 0/24 [00:00<?, ?it/s]

Hello, I also encountered the same problem, have you solved it?

@Furchtlos001
Copy link

@KaiLong1 I also encountered the same problem, have you solved it?

@superkoma
Copy link
Contributor

I have encountered this issue before, could you please provide the version of your installed waymo-open-dataset-tf, such as 2-6-0 or 2-0-0?

@zw-92
Copy link

zw-92 commented Sep 11, 2024

我以前遇到过这个问题,您能否提供您安装的版本,例如 或 ?waymo-open-dataset-tf``2-6-0``2-0-0

Hello, I used waymo-open-dataset-tf-2-5-0 1.4.1, but I found that there was a duplicate part of sequence_file_tfrecord = sequence_file[:-9] + '_with_camera_labels.tfrecord' in the waymo_utils.py, so I changed it to sequence_file_tfrecord= sequence_file[:-9] + '.tfrecord', but it gets stuck again in execution, log: CPU frequency: 3187200000 Hz
0%| | 0/48 [00:00<?, ?it/s], and I limited the number of processes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants