Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adapt image datasets #3362

Merged
merged 78 commits into from
Dec 9, 2021
Merged

Adapt image datasets #3362

merged 78 commits into from
Dec 9, 2021

Conversation

mariosasko
Copy link
Collaborator

@mariosasko mariosasko commented Dec 1, 2021 •
edited
Loading

This PR:

  • adapts the ImageClassification template to use the new Image feature
  • adapts the following datasets to use the new Image feature:
    • beans (+ fixes streaming)
    • cast_vs_dogs (+ fixes streaming)
    • cifar10
    • cifar100
    • fashion_mnist
    • mnist
    • head_qa

cc @nateraw

@mariosasko
Initial commit
42956f1
@mariosasko
Add basic decoding
93440cb
@mariosasko
Replace features.Audio with Audio
4baaae7
@mariosasko
Add Image to package reference
be25e24
@mariosasko
Use np.array
4660ce8
@mariosasko
Update error msg
b80633a
@mariosasko
Add mode and channel decoding
ac25393
@mariosasko
Fix return value
9946969
@mariosasko
Finish decoding
ff53e94
@mariosasko
Make CI happy
d1a91af
@mariosasko
Some more fixes
e628c9c
@mariosasko
Minor doc fix
bec851f
@mariosasko
Remove animated option
fab6e0f
@mariosasko
Pin version
539ffb5
@mariosasko
Remove unused imports in setup.py
924f94b
@mariosasko
Add vision requirements to setup.py
69eb82c
@mariosasko
Add initial tests
9a1a8ff
@mariosasko
Delete other formats
9d29246
@mariosasko
Make Image feature hashable
69b2f90
@mariosasko
Add more tests
50bee0a
@mariosasko
Support numpy array in alter data check in TypedSequence
4533c51
@mariosasko
Fix TypedSequence converion
1eec068
@mariosasko
Finish tests
3323d41
@mariosasko
Merge conflicts
bcc4ed4
@mariosasko
Update Image - add ImageExtensionType and supporting functions
db9a3eb
@mariosasko
Update encoding functions
d8e3ace
@mariosasko
Add support in TypedSequence for ImageExtensionType
f252dcb
@mariosasko
Add tests
c7b905d
@mariosasko
Merge branch 'master' of https://github.com/huggingface/datasets into…
d61e334 
… add-image-feature
@mariosasko
Remove unused import
d4d413a
mariosasko and others added 20 commits November 29, 2021 19:22
@mariosasko
Merge branch 'master' of https://github.com/huggingface/datasets into…
768b9f1 
… add-image-feature
@mariosasko
Test map that returns np.ndarray
544b093
@mariosasko
Rename functions
606215d
@mariosasko
Add streaming test
d7b204c
@mariosasko
Use image struct in all situations
31f02ad
@mariosasko @lhoestq
Update src/datasets/features/image.py - encode_example type hint
6ee758e 
Co-authored-by: Quentin Lhoest <42851186+lhoestq@users.noreply.github.com>
@mariosasko @lhoestq
Update src/datasets/features/image.py -list_image_compression_formats…
f294e5b 
… type hint

Co-authored-by: Quentin Lhoest <42851186+lhoestq@users.noreply.github.com>
@mariosasko
Use Image feature in ImageClassification task
0bd02b3
@mariosasko
Adapt cats_vs_dogs
5b7a9cf
@mariosasko
Adapt beans
ce0f5fe
@mariosasko
Adapt cifar10
17e57b3
@mariosasko
Adapt cifar100
815effb
@mariosasko
Add task templates to cifar10 and cifar100
b617ed9
@mariosasko
Adapt FashionMNIST
254aa43
@mariosasko
Adapt food101
9fbc083
@mariosasko
Adapt mnist datasets
195c3f2
@mariosasko
Adapt head_qa
ded7ebf
@mariosasko
Update example in head_qa readme
8c6de0b
@mariosasko
Update head_qa dummy data
917092c
@mariosasko
Fix streaming in beans and cats_vs_dogs
cc0d363
@mariosasko
Copy link
Collaborator Author

mariosasko commented Dec 6, 2021 •
edited
Loading

This PR can be merged after #3163 is merged (this PR is pretty big because I was working on the forked branch).

@lhoestq @albertvillanova Could you please take a look at the changes in src/datasets/utils/streaming_download_manager.py? These changes were required to support streaming of the cats_vs_dogs and the beans datasets.

@mariosasko mariosasko marked this pull request as ready for review December 7, 2021 14:24
@mariosasko
Copy link
Collaborator Author

The CI failures are due to the missing fields in the README files.

@mariosasko mariosasko closed this Dec 7, 2021
@mariosasko mariosasko reopened this Dec 7, 2021
lhoestq
lhoestq approved these changes Dec 9, 2021
View reviewed changes
Copy link
Member

@lhoestq lhoestq left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks all good to me ! feel free to merge if it's good on your side

@lhoestq
Copy link
Member

lhoestq commented Dec 9, 2021

and thanks for adding support for Path.name and Path.parent for streaming :)

@mariosasko mariosasko merged commit e855844 into master Dec 9, 2021
@mariosasko mariosasko deleted the adapt-image-datasets branch December 9, 2021 18:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lhoestq lhoestq lhoestq approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@mariosasko @lhoestq