-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Minor changes for 4.1 in tub conversion script and developer doc #708
Changes from 1 commit
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
* Update conversion script to translate discontinuous data. * In developer guide add disclaimer for version and correct intra-page links.
- Loading branch information
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -22,29 +22,44 @@ | |
|
||
|
||
def convert_to_tub_v2(paths, output_path): | ||
if type(paths) is str: | ||
paths = [paths] | ||
legacy_tubs = [LegacyTub(path) for path in paths] | ||
output_tub = None | ||
print('Total number of tubs: %s' % (len(legacy_tubs))) | ||
print(f'Total number of tubs: {len(legacy_tubs)}') | ||
|
||
for legacy_tub in legacy_tubs: | ||
if not output_tub: | ||
output_tub = Tub(output_path, legacy_tub.inputs, legacy_tub.types, list(legacy_tub.meta.items())) | ||
output_tub = Tub(output_path, legacy_tub.inputs, | ||
legacy_tub.types, list(legacy_tub.meta.items())) | ||
|
||
record_paths = legacy_tub.gather_records() | ||
bar = IncrementalBar('Converting', max=len(record_paths)) | ||
|
||
prev_rec_num = None | ||
for record_path in record_paths: | ||
try: | ||
contents = Path(record_path).read_text() | ||
record = json.loads(contents) | ||
image_path = record['cam/image_array'] | ||
rec_num = int(image_path.split('_')[0]) | ||
DocGarbanzo marked this conversation as resolved.
Show resolved
Hide resolved
|
||
image_path = os.path.join(legacy_tub.path, image_path) | ||
image_data = Image.open(image_path) | ||
record['cam/image_array'] = image_data | ||
output_tub.write_record(record) | ||
# first record or they are continuous, just append | ||
if not prev_rec_num or rec_num == prev_rec_num + 1: | ||
output_tub.write_record(record) | ||
prev_rec_num = rec_num | ||
# otherwise fill the gap with dummy records | ||
else: | ||
prev_rec_num += 1 | ||
DocGarbanzo marked this conversation as resolved.
Show resolved
Hide resolved
|
||
while prev_rec_num < rec_num: | ||
idx = output_tub.manifest.current_index | ||
output_tub.write_record({}) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Define a {'__empty__': true} Write that for every deleted record. That way the intention here is very clear when reading though the file. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good point. I first thought we might want to support such field in the tub itself, but not sure... I 've updated the script to handle this. |
||
output_tub.delete_record(idx) | ||
prev_rec_num += 1 | ||
bar.next() | ||
except Exception as exception: | ||
print('Ignoring record path %s\n' % (record_path), exception) | ||
print(f'Ignoring record path {record_path}\n', exception) | ||
traceback.print_exc() | ||
|
||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: Call this
previous_index
.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok.