Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[azure] Allow creation of signed URLs for upload #1414

Merged
merged 3 commits into from
Jul 5, 2024

Conversation

CorentinGoodays
Copy link
Contributor

Allow creation of signed URLs for upload in AzureStorage (See #1413)

@jschneier
Copy link
Owner

Thanks for the PR, looks okay. Two comments:

  1. Is there a library version minimum for the from_string api?
  2. Can you please add or update a test case that covers this change

@CorentinGoodays
Copy link
Contributor Author

CorentinGoodays commented Jun 30, 2024

Thank you for reviewing my PR and for your feedback.

According to the release notes, the BlobSasPermissions.from_string method was introduced in version 12.0.0b4:

New features:
...
AccountSasPermissions, BlobSasPermissions, ContainerSasPermissions now have method from_string which takes parameters as a string.

This version was released a few weeks before version 12.0.0, so from my understanding, there is no need to change the requirements of django-storages.

Regarding the test case, I believe the modification I'm proposing is already covered by this existing test:
https://github.com/jschneier/django-storages/blob/master/tests/test_azure.py#L157

This test currently only checks the functionality with read permissions, but there is no functional difference between a signed URL with read, write, or any other permissions. The only difference resides in the query string of the resulting URL. There's is a sp parameter which contains the permission string (like r, or w etc), and the sig parameter which contains the actual signature. Since the query string is not included in the mock implementation, I am not certain what additional aspects could be tested.

@jschneier
Copy link
Owner

Sounds good on point (1).

For point (2), we should update the test or tweak it such that we confirm this is covered both for the default case and at least for the write case. I want the test to break if someone accidentally reverts parts of your patch.

@CorentinGoodays
Copy link
Contributor Author

CorentinGoodays commented Jul 3, 2024

I updated the test to cover both the default case and the write case.

Initially, I was hoping I could do this:

            generate_blob_sas_mocked.assert_called_once_with(
                self.account_name,
                self.container_name,
                "some blob",
                account_key=self.account_key,
                user_delegation_key=None,
-               permission=mock.ANY,
+               permission=BlobSasPermission(read=True),
                expiry=fixed_time + timedelta(seconds=100),
            )
...
            generate_blob_sas_mocked.assert_called_with(
                self.account_name,
                self.container_name,
                "some blob",
                account_key=self.account_key,
                user_delegation_key=None,
-               permission=mock.ANY,
+               permission=BlobSasPermission(write=True),
                expiry=fixed_time + timedelta(seconds=100),
            )

But sadly Microsoft has not implemented the __eq__ method for the BlobSasPermissions class, so the check is not working because it's comparing two different instances :/

I resorted to this instead:

            called_args, called_kwargs = generate_blob_sas_mocked.call_args
            self.assertEqual(str(called_kwargs["permission"]), "r")
...
            called_args, called_kwargs = generate_blob_sas_mocked.call_args
            self.assertEqual(str(called_kwargs["permission"]), "w")

Are you OK with this approach?

Would you prefer to have a separate test case for the write permission?

@jschneier
Copy link
Owner

Ah that’s too bad. Is there a property on the class we can check?

@CorentinGoodays
Copy link
Contributor Author

There's a _str field whose value is based on the boolean values (read, write, etc) passed to the constructor. That's actually what I'm checking by converting the permission to a string here:

self.assertEqual(str(called_kwargs["permission"]), "w")

An alternative could be to dynamically add an __eq__ method to the BlobSasPermissions class, like this:

# Dynamically add __eq__ method to BlobSasPermissions
def blob_sas_permissions_eq(self, other):
    if not isinstance(other, BlobSasPermissions):
        return NotImplemented
    return self._str == other._str

BlobSasPermissions.__eq__ = blob_sas_permissions_eq

Then this code would work as expected:

            generate_blob_sas_mocked.assert_called_with(
                self.account_name,
                self.container_name,
                "some blob",
                account_key=self.account_key,
                user_delegation_key=None,
                permission=BlobSasPermission(write=True),
                expiry=fixed_time + timedelta(seconds=100),
            )

@jschneier
Copy link
Owner

Thanks for updating, looks great!

@jschneier
Copy link
Owner

Can you just fix the lint error? Good to merge after that.

@jschneier
Copy link
Owner

Also if you wouldn't mind adding to the CHANGELOG file, that'd be appreciated!

diff --git a/CHANGELOG.rst b/CHANGELOG.rst
index 6cdc419..0b0085f 100644
--- a/CHANGELOG.rst
+++ b/CHANGELOG.rst
@@ -15,11 +15,13 @@ Azure
 -----
 
 - Fix ``collectstatic --clear`` (`#1403`_)
+- Add ``mode`` kwarg to ``.url()`` to support creation of signed URLs for upload (`#1414`_)
 
 .. _#1399: https://github.com/jschneier/django-storages/pull/1399
 .. _#1381: https://github.com/jschneier/django-storages/pull/1381
 .. _#1402: https://github.com/jschneier/django-storages/pull/1402
 .. _#1403: https://github.com/jschneier/django-storages/pull/1403
+.. _#1414: https://github.com/jschneier/django-storages/pull/1414

CorentinGoodays added a commit to CorentinGoodays/django-storages that referenced this pull request Jul 5, 2024
@CorentinGoodays
Copy link
Contributor Author

All right, I updated the changelog and addressed the lint error (though I couldn't reproduce it using tox, only when running Black 23.3.0 manually).

I'm not clear whether you're satisfied with this way of doing:

called_args, called_kwargs = generate_blob_sas_mocked.call_args
self.assertEqual(str(called_kwargs["permission"]), "w")

or if you would prefer adding the __eq__ method to BlobSasPermissions like this?

# Dynamically add __eq__ method to BlobSasPermissions
def blob_sas_permissions_eq(self, other):
    if not isinstance(other, BlobSasPermissions):
        return NotImplemented
    return self._str == other._str

BlobSasPermissions.__eq__ = blob_sas_permissions_eq

...
            generate_blob_sas_mocked.assert_called_with(
                self.account_name,
                self.container_name,
                "some blob",
                account_key=self.account_key,
                user_delegation_key=None,
                permission=BlobSasPermission(write=True),
                expiry=fixed_time + timedelta(seconds=100),
            )

I think the second approach is nice as it makes the test case more readable. Additionally, if Microsoft ever implements the __eq__ method, we can easily remove our custom implementation.

Please let me know which approach you prefer.

@jschneier jschneier linked an issue Jul 5, 2024 that may be closed by this pull request
@jschneier jschneier merged commit 8271347 into jschneier:master Jul 5, 2024
19 checks passed
ikarius referenced this pull request in gip-inclusion/dora-back Jul 29, 2024
Bumps
[django-storages[boto3]](https://github.com/jschneier/django-storages)
from 1.14.3 to 1.14.4.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/jschneier/django-storages/blob/master/CHANGELOG.rst">django-storages[boto3]'s
changelog</a>.</em></p>
<blockquote>
<p>1.14.4 (2024-07-09)</p>
<hr />
<h2>S3</h2>
<ul>
<li>Pull <code>AWS_SESSION_TOKEN</code> from the environment
(<code>[#1399](https://github.com/jschneier/django-storages/issues/1399)</code>_)</li>
<li>Fix newline handling for text mode files
(<code>[#1381](https://github.com/jschneier/django-storages/issues/1381)</code>_)</li>
<li>Do not sign URLs when <code>querystring_auth=False</code> e.g public
buckets or static files
(<code>[#1402](https://github.com/jschneier/django-storages/issues/1402)</code>_)</li>
<li>Cache CloudFront Signers
(<code>[#1417](https://github.com/jschneier/django-storages/issues/1417)</code>_)</li>
</ul>
<h2>Azure</h2>
<ul>
<li>Fix <code>collectstatic --clear</code>
(<code>[#1403](https://github.com/jschneier/django-storages/issues/1403)</code>_)</li>
<li>Add <code>mode</code> kwarg to <code>.url()</code> to support
creation of signed URLs for upload
(<code>[#1414](https://github.com/jschneier/django-storages/issues/1414)</code>_)</li>
<li>Fix fetching user delegation key when custom domain is enabled
(<code>[#1418](https://github.com/jschneier/django-storages/issues/1418)</code>_)</li>
</ul>
<h2>SFTP</h2>
<ul>
<li>Add implementations of <code>get_(modified|accessed)_time</code>
(<code>[#1347](https://github.com/jschneier/django-storages/issues/1347)</code>_)</li>
</ul>
<h2>Dropbox</h2>
<ul>
<li>Add support for Python 3.12
(<code>[#1421](https://github.com/jschneier/django-storages/issues/1421)</code>_)</li>
</ul>
<h2>FTP</h2>
<ul>
<li>Conform to <code>BaseStorage</code> interface
(<code>[#1423](https://github.com/jschneier/django-storages/issues/1423)</code>_)</li>
<li>Add <code>FTP_ALLOW_OVERWRITE</code> setting
(<code>[#1424](https://github.com/jschneier/django-storages/issues/1424)</code>_)</li>
</ul>
<p>.. _<a
href="https://redirect.github.com/jschneier/django-storages/issues/1399">#1399</a>:
<a
href="https://redirect.github.com/jschneier/django-storages/pull/1399">jschneier/django-storages#1399</a>
.. _<a
href="https://redirect.github.com/jschneier/django-storages/issues/1381">#1381</a>:
<a
href="https://redirect.github.com/jschneier/django-storages/pull/1381">jschneier/django-storages#1381</a>
.. _<a
href="https://redirect.github.com/jschneier/django-storages/issues/1402">#1402</a>:
<a
href="https://redirect.github.com/jschneier/django-storages/pull/1402">jschneier/django-storages#1402</a>
.. _<a
href="https://redirect.github.com/jschneier/django-storages/issues/1403">#1403</a>:
<a
href="https://redirect.github.com/jschneier/django-storages/pull/1403">jschneier/django-storages#1403</a>
.. _<a
href="https://redirect.github.com/jschneier/django-storages/issues/1414">#1414</a>:
<a
href="https://redirect.github.com/jschneier/django-storages/pull/1414">jschneier/django-storages#1414</a>
.. _<a
href="https://redirect.github.com/jschneier/django-storages/issues/1417">#1417</a>:
<a
href="https://redirect.github.com/jschneier/django-storages/pull/1417">jschneier/django-storages#1417</a>
.. _<a
href="https://redirect.github.com/jschneier/django-storages/issues/1418">#1418</a>:
<a
href="https://redirect.github.com/jschneier/django-storages/pull/1418">jschneier/django-storages#1418</a>
.. _<a
href="https://redirect.github.com/jschneier/django-storages/issues/1347">#1347</a>:
<a
href="https://redirect.github.com/jschneier/django-storages/pull/1347">jschneier/django-storages#1347</a>
.. _<a
href="https://redirect.github.com/jschneier/django-storages/issues/1421">#1421</a>:
<a
href="https://redirect.github.com/jschneier/django-storages/pull/1421">jschneier/django-storages#1421</a>
.. _<a
href="https://redirect.github.com/jschneier/django-storages/issues/1423">#1423</a>:
<a
href="https://redirect.github.com/jschneier/django-storages/pull/1423">jschneier/django-storages#1423</a>
.. _<a
href="https://redirect.github.com/jschneier/django-storages/issues/1424">#1424</a>:
<a
href="https://redirect.github.com/jschneier/django-storages/pull/1424">jschneier/django-storages#1424</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/jschneier/django-storages/commit/b37d53e1209f6333de71502a9cb34d8cffdc7e9f"><code>b37d53e</code></a>
Release version 1.14.4 (<a
href="https://redirect.github.com/jschneier/django-storages/issues/1428">#1428</a>)</li>
<li><a
href="https://github.com/jschneier/django-storages/commit/a79a304e10917c1b8e64cb3fcb3c3ea52c049718"><code>a79a304</code></a>
[sftp] Cleanup duplicated properties (<a
href="https://redirect.github.com/jschneier/django-storages/issues/1426">#1426</a>)</li>
<li><a
href="https://github.com/jschneier/django-storages/commit/cd42c1720247dd9c363ac99a5fa97ef4cf2e4acb"><code>cd42c17</code></a>
[dropbox] Remove dead prefix removal code (<a
href="https://redirect.github.com/jschneier/django-storages/issues/1425">#1425</a>)</li>
<li><a
href="https://github.com/jschneier/django-storages/commit/855e58dd8a6a0d6cbc2d5db70f6566ea89ba2bd4"><code>855e58d</code></a>
[ftp] Add FTP_ALLOW_OVERWRITE setting (<a
href="https://redirect.github.com/jschneier/django-storages/issues/1424">#1424</a>)</li>
<li><a
href="https://github.com/jschneier/django-storages/commit/fe759bffedf6462b478ab2bafd973038e821a3b1"><code>fe759bf</code></a>
[ftp] Conform to BaseStorage interface (<a
href="https://redirect.github.com/jschneier/django-storages/issues/1423">#1423</a>)</li>
<li><a
href="https://github.com/jschneier/django-storages/commit/a5319477473e5bab60b0d698e2a8127a8f750787"><code>a531947</code></a>
[general] Write overwriting files in terms of .exists() (<a
href="https://redirect.github.com/jschneier/django-storages/issues/1422">#1422</a>)</li>
<li><a
href="https://github.com/jschneier/django-storages/commit/0041eb205dc7476210cc7c263574e807322d33d6"><code>0041eb2</code></a>
[dropbox] Add support for Python3.12 (<a
href="https://redirect.github.com/jschneier/django-storages/issues/1421">#1421</a>)</li>
<li><a
href="https://github.com/jschneier/django-storages/commit/4cd12e088bf1be39e49c20f7fa1d1e7f7ed7c61b"><code>4cd12e0</code></a>
[sftp] Format times to UTC (<a
href="https://redirect.github.com/jschneier/django-storages/issues/1420">#1420</a>)</li>
<li><a
href="https://github.com/jschneier/django-storages/commit/ea2522602492fae494f68511f864dc8dd0266098"><code>ea25226</code></a>
[sftp] Add implementations of get_(modified|accessed)_time (<a
href="https://redirect.github.com/jschneier/django-storages/issues/1347">#1347</a>)</li>
<li><a
href="https://github.com/jschneier/django-storages/commit/6c494a071a907b75727a15898167a27cd0d2cb89"><code>6c494a0</code></a>
[docs/scaleway] Create Scaleway docs (<a
href="https://redirect.github.com/jschneier/django-storages/issues/1419">#1419</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/jschneier/django-storages/compare/1.14.3...1.14.4">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=django-storages[boto3]&package-manager=pip&previous-version=1.14.3&new-version=1.14.4)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: ikarius <fred@ikarius.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

AzureStorage does not allow creation of signed URLs for upload
2 participants