Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Start setting up the improved W&B integration #1948

Merged
merged 36 commits into from
Feb 2, 2021
Merged
Changes from 1 commit
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
3530aa5
Add helper functions for wandb and artifacts
AyushExel Jan 15, 2021
1ef16b1
cleanup
AyushExel Jan 15, 2021
bd37ac7
Reorganize files
AyushExel Jan 15, 2021
7cd5623
Update wandb_utils.py
glenn-jocher Jan 17, 2021
d8daa64
Update log_dataset.py
glenn-jocher Jan 17, 2021
fc7d8f9
Reorganize and update dataloader call
AyushExel Jan 18, 2021
9b8f46a
yaml.SafeLoader
glenn-jocher Jan 20, 2021
2c67ba1
PEP8 reformat
glenn-jocher Jan 20, 2021
9ffb887
remove redundant checks
glenn-jocher Jan 20, 2021
1eca722
Add helper functions for wandb and artifacts
AyushExel Jan 15, 2021
ad8408f
cleanup
AyushExel Jan 15, 2021
8b039b3
Reorganize files
AyushExel Jan 15, 2021
0b715eb
Update wandb_utils.py
glenn-jocher Jan 17, 2021
58ae23f
Update log_dataset.py
glenn-jocher Jan 17, 2021
6671347
Reorganize and update dataloader call
AyushExel Jan 18, 2021
bbc5271
yaml.SafeLoader
glenn-jocher Jan 20, 2021
9bbecc8
PEP8 reformat
glenn-jocher Jan 20, 2021
ceb63cd
remove redundant checks
glenn-jocher Jan 20, 2021
2529fb7
Update util files
AyushExel Jan 21, 2021
efdcd7e
Update wandb_utils.py
AyushExel Jan 21, 2021
0d84870
Remove word size
AyushExel Jan 21, 2021
93219b9
Change path of labels.zip
AyushExel Jan 21, 2021
ac8fad2
remove unused imports
glenn-jocher Jan 23, 2021
e40b817
remove --rect
glenn-jocher Jan 23, 2021
0e21989
log_dataset.py cleanup
glenn-jocher Jan 23, 2021
3e53865
log_dataset.py cleanup2
glenn-jocher Jan 23, 2021
4ebfb83
wandb_utils.py cleanup
glenn-jocher Jan 23, 2021
58616a6
remove redundant id_count
glenn-jocher Jan 23, 2021
a8168c3
wandb_utils.py cleanup2
glenn-jocher Jan 23, 2021
1bb261d
rename cls
glenn-jocher Jan 23, 2021
850b59d
use pathlib for zip
glenn-jocher Jan 23, 2021
94b7631
rename dataloader to dataset
glenn-jocher Jan 23, 2021
b959a02
Change import order
AyushExel Jan 27, 2021
4e9a54c
Remove redundant code
AyushExel Jan 30, 2021
28ad4ab
remove unused import
AyushExel Jan 30, 2021
5651225
remove unused imports
glenn-jocher Feb 2, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
use pathlib for zip
  • Loading branch information
glenn-jocher authored and AyushExel committed Feb 1, 2021
commit 850b59da1a19409a9d17ad89ca66f3aab92a719f
11 changes: 5 additions & 6 deletions utils/wandb_logging/wandb_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -120,10 +120,9 @@ def log_dataset_artifact(self, dataloader, class_to_id, name='dataset'):
labels[:, 2:] = (xywh2xyxy(labels[:, 2:].view(-1, 4)))
height, width = shapes[0]
labels[:, 2:] *= torch.Tensor([width, height, width, height])
labels = labels[:, 1:]
box_data = []
img_classes = {}
for cls, *xyxy in labels.tolist():
for cls, *xyxy in labels[:, 1:].tolist():
cls = int(cls)
box_data.append({"position": {"minX": xyxy[0], "minY": xyxy[1], "maxX": xyxy[2], "maxY": xyxy[3]},
"class_id": cls,
Expand All @@ -135,10 +134,10 @@ def log_dataset_artifact(self, dataloader, class_to_id, name='dataset'):
table.add_data(si, wandb.Image(paths, classes=class_set, boxes=boxes), json.dumps(img_classes))
artifact.add(table, name)
labels_path = image_path.replace('images', 'labels')
labels_zipped_path = Path(labels_path).parent / (name + '_labels.zip')
if not os.path.isfile(labels_zipped_path): # make_archive won't check if file exists
shutil.make_archive(Path(labels_path).parent / (name + '_labels'), 'zip', labels_path)
artifact.add_file(str(labels_zipped_path), name='data/labels.zip')
zip_path = Path(labels_path).parent / (name + '_labels.zip')
if not zip_path.is_file(): # make_archive won't check if file exists
shutil.make_archive(zip_path.with_suffix(''), 'zip', labels_path)
artifact.add_file(str(zip_path), name='data/labels.zip')
wandb.log_artifact(artifact)
print("Saving data to W&B...")

Expand Down