Skip to content
This repository has been archived by the owner on Dec 16, 2022. It is now read-only.

Make images easier to find for Visual Entailment #4878

Merged
merged 73 commits into from
Dec 21, 2020
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
f1ed56a
implement TorchImageLoader
epwalsh Nov 25, 2020
2ff1488
implement ResnetBackbone
epwalsh Nov 26, 2020
17df0df
add resize + normalize to image loader
epwalsh Nov 30, 2020
2395fec
finalize FasterRcnnRegionDetector
epwalsh Dec 1, 2020
cdf5090
pin torchvision
epwalsh Dec 1, 2020
850735c
fix VQAv2Reader
epwalsh Dec 1, 2020
037da33
add box mask field
epwalsh Dec 2, 2020
12979cc
fix merge conflicts
epwalsh Dec 2, 2020
99e5928
dataset reader fixes
epwalsh Dec 2, 2020
38499d2
fix model tests
epwalsh Dec 2, 2020
a2de991
doc fixes
epwalsh Dec 2, 2020
cbd5800
fix merge conflicts
epwalsh Dec 3, 2020
ae4c839
add threshold parameters to FasterRcnnRegionDetector
epwalsh Dec 3, 2020
1252879
address @dirkgr comments
epwalsh Dec 4, 2020
d5bd779
mask fixes
epwalsh Dec 4, 2020
f3df7a2
shape comments
epwalsh Dec 4, 2020
877b7b5
add some more comments
epwalsh Dec 4, 2020
c491536
cache answers_by_question_id
epwalsh Dec 5, 2020
7007fb4
implement LocalCacheResource
epwalsh Dec 5, 2020
38c32ee
fix
epwalsh Dec 5, 2020
4b33a63
Merge branch 'vision' into torchvision
epwalsh Dec 5, 2020
3376d08
add read-only option to cache
epwalsh Dec 6, 2020
415b6ef
fix
epwalsh Dec 6, 2020
a603611
simplify data loader
epwalsh Dec 7, 2020
797e95e
make featurizer and detector optional in readers
epwalsh Dec 7, 2020
a4baae6
Cache in memory
dirkgr Dec 8, 2020
18ca15f
back pressure is important I guess
epwalsh Dec 9, 2020
625f6f3
Merge branch 'torchvision' of github.com:allenai/allennlp into torchv…
epwalsh Dec 9, 2020
0aa3331
merge
epwalsh Dec 9, 2020
273b453
Updated configs
dirkgr Dec 9, 2020
0a206f3
Fixes the way we apply masks
dirkgr Dec 9, 2020
f806eb7
Merge branch 'torchvision' of https://github.com/allenai/allennlp int…
dirkgr Dec 9, 2020
2f5ebf1
Use more of Jiasen's real settings
dirkgr Dec 9, 2020
48c7620
Upgrade the from_huggingface config
dirkgr Dec 9, 2020
f031d5e
Switch back to the images on corpnet
dirkgr Dec 9, 2020
23c3986
Fix random seeds
dirkgr Dec 9, 2020
8dde832
Bigger model needs smaller batch size
dirkgr Dec 9, 2020
8266b83
Adds ability to selectively ignore one input
dirkgr Dec 9, 2020
0f8faac
address some comments
epwalsh Dec 14, 2020
83292d7
Merge branch 'vision' into torchvision
epwalsh Dec 14, 2020
dd4dff8
format + lint
epwalsh Dec 14, 2020
de7f706
fixes
epwalsh Dec 14, 2020
945896f
Bring back bert-base configs
dirkgr Dec 15, 2020
89b98e3
Merge branch 'torchvision' of https://github.com/allenai/allennlp int…
dirkgr Dec 15, 2020
e598453
Merge branch 'vision' into torchvision
dirkgr Dec 15, 2020
c06b0b3
Merge branch 'vision' into torchvision
dirkgr Dec 15, 2020
2303d6e
Merge branch 'vision' into torchvision
dirkgr Dec 15, 2020
55b3c4f
Merge branch 'vision' into torchvision
dirkgr Dec 16, 2020
7af271f
fix error handling
epwalsh Dec 16, 2020
1bf7fc7
fix test
epwalsh Dec 16, 2020
962c965
Merge branch 'vision' into torchvision
epwalsh Dec 16, 2020
490b909
Adds the ability to read from a feature cache, but not run any featur…
dirkgr Dec 17, 2020
edf57f4
Update tests
dirkgr Dec 17, 2020
a9bc3da
Let's stick with "feature_cache"
dirkgr Dec 17, 2020
a5e958d
More epochs, more random
dirkgr Dec 17, 2020
12ba53c
Use the new parameters
dirkgr Dec 17, 2020
14f8a0b
Fix initialization
dirkgr Dec 17, 2020
87647a5
Merge remote-tracking branch 'origin/vision' into NewFeatures
dirkgr Dec 17, 2020
5e9b393
Make tests work, add some documentation
dirkgr Dec 18, 2020
732dab2
Remove the read_from_cache parameter
dirkgr Dec 18, 2020
c84be68
Cleanup of training configs
dirkgr Dec 18, 2020
dbfa73b
Typecheck
dirkgr Dec 18, 2020
ec41a70
Building docs right
dirkgr Dec 18, 2020
87e4087
Better settings for VQA
dirkgr Dec 19, 2020
9aadaa4
Merge branch 'NewFeatures' of https://github.com/allenai/allennlp int…
dirkgr Dec 19, 2020
2565490
Open cached paths when reading json lines
dirkgr Dec 19, 2020
4d16e3f
By default, autodetect GPUs when training
dirkgr Dec 19, 2020
e1f31b7
Switch to torchvision
dirkgr Dec 19, 2020
dab8370
Download training data from the web
dirkgr Dec 19, 2020
33f9723
Merge remote-tracking branch 'origin/vision' into VEPaths
dirkgr Dec 21, 2020
31e866d
This needs to stay at 1024 until we get the new featurization model
dirkgr Dec 21, 2020
4dd7a8f
Have a more descriptive error message when images are missing
dirkgr Dec 21, 2020
4e9c8ea
Update vilbert_ve_from_huggingface.jsonnet
AkshitaB Dec 21, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
address @dirkgr comments
  • Loading branch information
epwalsh committed Dec 4, 2020
commit 12528797b43d9fd65bbe82adf1a8dd22cc77af11
4 changes: 2 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,11 @@ COPY allennlp/version.py allennlp/version.py
COPY setup.py .
RUN touch allennlp/__init__.py \
&& touch README.md \
&& pip install --no-cache-dir -e .[vision]
&& pip install --no-cache-dir -e .

# Now add the full package source and re-install just the package.
COPY allennlp allennlp
RUN pip install --no-cache-dir --no-deps -e .[vision]
RUN pip install --no-cache-dir --no-deps -e .

WORKDIR /app/

Expand Down
4 changes: 2 additions & 2 deletions Dockerfile.test
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,10 @@ COPY setup.py .
COPY dev-requirements.txt .
RUN touch allennlp/__init__.py \
&& touch README.md \
&& pip install --no-cache-dir -e .[vision] -r dev-requirements.txt
&& pip install --no-cache-dir -e . -r dev-requirements.txt

# Now add the full package source and re-install just the package.
COPY . .
RUN pip install --no-cache-dir --no-deps -e .[vision]
RUN pip install --no-cache-dir --no-deps -e .

ENTRYPOINT ["make"]
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ install :
# Due to a weird thing with pip, we may need egg-info before running `pip install -e`.
# See https://github.com/pypa/pip/issues/4537.
python setup.py install_egg_info
pip install --upgrade --upgrade-strategy eager -e .[vision] -r dev-requirements.txt
pip install --upgrade --upgrade-strategy eager -e . -r dev-requirements.txt

#
# Documention helpers.
Expand Down
3 changes: 1 addition & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,8 +151,7 @@ to distribute as a plugin, see the [subcommand API docs](https://docs.allennlp.o
AllenNLP requires Python 3.6.1 or later and [PyTorch](https://pytorch.org/).
It's recommended that you install the PyTorch ecosystem **before** installing AllenNLP by following the instructions on [pytorch.org](https://pytorch.org/).

The preferred way to install AllenNLP is via `pip`. Just run `pip install allennlp`. Or, if you want the vision
features of the library, run `pip install allennlp[vision]`.
The preferred way to install AllenNLP is via `pip`. Just run `pip install allennlp`.

> ⚠️ If you're using Python 3.7 or greater, you should ensure that you don't have the PyPI version of `dataclasses` installed after running the above command, as this could cause issues on certain platforms. You can quickly check this by running `pip freeze | grep dataclasses`. If you see something like `dataclasses=0.6` in the output, then just run `pip uninstall -y dataclasses`.

Expand Down
12 changes: 1 addition & 11 deletions allennlp/common/util.py
Original file line number Diff line number Diff line change
Expand Up @@ -342,17 +342,7 @@ def import_module_and_submodules(package_name: str) -> None:
# the end won't hurt anything.
with push_python_path("."):
# Import at top level
try:
module = importlib.import_module(package_name)
except ImportError as err:
if err.name in {"torchvision"}:
logger.warning(
"vision module '%s' is unavailable since '%s' is not installed",
package_name,
err.name,
)
return None
raise
module = importlib.import_module(package_name)
path = getattr(module, "__path__", [])
path_string = "" if not path else path[0]

Expand Down
7 changes: 1 addition & 6 deletions allennlp/data/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,4 @@
from allennlp.data.tokenizers.tokenizer import Tokenizer
from allennlp.data.vocabulary import Vocabulary
from allennlp.data.batch import Batch

try:
from allennlp.data.image_loader import ImageLoader, TorchImageLoader
except ModuleNotFoundError as err:
if err.name not in {"torchvision"}:
raise
from allennlp.data.image_loader import ImageLoader, TorchImageLoader
13 changes: 4 additions & 9 deletions allennlp/data/dataset_readers/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,7 @@
from allennlp.data.dataset_readers.sequence_tagging import SequenceTaggingDatasetReader
from allennlp.data.dataset_readers.sharded_dataset_reader import ShardedDatasetReader
from allennlp.data.dataset_readers.text_classification_json import TextClassificationJsonReader

try:
from allennlp.data.dataset_readers.vision_reader import VisionReader
from allennlp.data.dataset_readers.vqav2 import VQAv2Reader
from allennlp.data.dataset_readers.visual_entailment import VisualEntailmentReader
from allennlp.data.dataset_readers.gqa import GQAReader
except ModuleNotFoundError as err:
if err.name not in {"torchvision"}:
raise
from allennlp.data.dataset_readers.vision_reader import VisionReader
from allennlp.data.dataset_readers.vqav2 import VQAv2Reader
from allennlp.data.dataset_readers.visual_entailment import VisualEntailmentReader
from allennlp.data.dataset_readers.gqa import GQAReader
2 changes: 1 addition & 1 deletion allennlp/data/dataset_readers/gqa.py
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ def text_to_instance(
"box_features": ArrayField(features),
"box_coordinates": ArrayField(coords),
"box_mask": ArrayField(
features.new_full((features.shape[0],), True, dtype=torch.bool),
features.new_ones((features.shape[0],), dtype=torch.bool),
padding_value=False,
dtype=torch.bool,
),
Expand Down
2 changes: 1 addition & 1 deletion allennlp/data/dataset_readers/visual_entailment.py
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ def text_to_instance(
fields["box_features"] = ArrayField(features)
fields["box_coordinates"] = ArrayField(coords)
fields["box_mask"] = ArrayField(
features.new_full((features.shape[0],), True, dtype=torch.bool),
features.new_ones((features.shape[0],), dtype=torch.bool),
padding_value=False,
dtype=torch.bool,
)
Expand Down
2 changes: 1 addition & 1 deletion allennlp/data/dataset_readers/vqav2.py
Original file line number Diff line number Diff line change
Expand Up @@ -513,7 +513,7 @@ def text_to_instance(
fields["box_features"] = ArrayField(features)
fields["box_coordinates"] = ArrayField(coords)
fields["box_mask"] = ArrayField(
features.new_full((features.shape[0],), True, dtype=torch.bool),
features.new_ones((features.shape[0],), dtype=torch.bool),
padding_value=False,
dtype=torch.bool,
)
Expand Down
4 changes: 3 additions & 1 deletion allennlp/data/image_loader.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ class ImageLoader(Registrable):

def __init__(
self,
*,
size_divisibility: int = 0,
pad_value: float = 0.0,
device: Union[str, torch.device] = "cpu",
Expand Down Expand Up @@ -146,6 +147,7 @@ class TorchImageLoader(ImageLoader):

def __init__(
self,
*,
image_backend: str = None,
resize: bool = True,
normalize: bool = True,
Expand All @@ -154,7 +156,7 @@ def __init__(
pixel_mean: Tuple[float, float, float] = (0.485, 0.456, 0.406),
pixel_std: Tuple[float, float, float] = (0.229, 0.224, 0.225),
size_divisibility: int = 32,
**kwargs
**kwargs,
) -> None:
super().__init__(size_divisibility=size_divisibility, **kwargs)
if image_backend is not None:
Expand Down
1 change: 1 addition & 0 deletions allennlp/modules/vision/region_detector.py
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,7 @@ class FasterRcnnRegionDetector(RegionDetector):

def __init__(
self,
*,
box_score_thresh: float = 0.05,
box_nms_thresh: float = 0.5,
max_boxes_per_image: int = 100,
Expand Down
6 changes: 1 addition & 5 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@
),
install_requires=[
"torch>=1.6.0,<1.8.0",
"torchvision>=0.8.1,<0.9.0",
"jsonnet>=0.10.0 ; sys.platform != 'win32'",
"overrides==3.1.0",
"nltk",
Expand All @@ -71,11 +72,6 @@
"lmdb",
"more-itertools",
],
extras_require={
"vision": [
"torchvision>=0.8.1,<0.9.0",
],
},
entry_points={"console_scripts": ["allennlp=allennlp.__main__:run"]},
include_package_data=True,
python_requires=">=3.6.1",
Expand Down