Skip to content
/ beam Public
forked from apache/beam

Commit

Permalink
Support dict as the input of delete_batch (apache#30188)
Browse files Browse the repository at this point in the history
This smooths the code path of feeding output of list_prefix()
to delete_batch().
  • Loading branch information
shunping committed Feb 1, 2024
1 parent bdbbef9 commit 1a5466b
Showing 1 changed file with 3 additions and 1 deletion.
4 changes: 3 additions & 1 deletion sdks/python/apache_beam/io/gcp/gcsio.py
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,8 @@ def delete_batch(self, paths):
"""Deletes the objects at the given GCS paths.
Args:
paths: List of GCS file path patterns in the form gs://<bucket>/<name>,
paths: List of GCS file path patterns or Dict with GCS file path patterns
as keys. The patterns are in the form gs://<bucket>/<name>, but
not to exceed MAX_BATCH_OPERATION_SIZE in length.
Returns: List of tuples of (path, exception) in the same order as the
Expand All @@ -217,6 +218,7 @@ def delete_batch(self, paths):
"""
final_results = []
s = 0
if not isinstance(paths, list): paths = list(iter(paths))
while s < len(paths):
if (s + MAX_BATCH_OPERATION_SIZE) < len(paths):
current_paths = paths[s:s + MAX_BATCH_OPERATION_SIZE]
Expand Down

0 comments on commit 1a5466b

Please sign in to comment.