galaxy.yml: exclude non-collection files from collection build #3517

dmsimard · 2021-10-05T20:56:25Z

SUMMARY

Files that belong to the git repository such as git, github and
azure pipeline configuration should not be included in the published
collection tarball.

Excluding tests saves on what makes the bulk of the size of the
collection, both in size and number of files.

In aggregate, excluding these files results in a meaningful improvement
which will speed up the installation of the collection and reduce the
size on disk.

ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME

galaxy.yml

ADDITIONAL INFORMATION

Before

> du -sh community-general-3.7.0.tar.gz 
2.2M community-general-3.7.0.tar.gz

> du -sh community.general
18M	community.general

> find . -type f | wc -l
2127

> time ansible-galaxy collection install /home/dmsimard/Downloads/community-general-3.7.0.tar.gz 
Starting galaxy collection install process
Process install dependency map
Starting collection install process
Installing 'community.general:3.7.0' to '/home/dmsimard/.ansible/collections/ansible_collections/community/general'
community.general:3.7.0 was installed successfully
________________________________________________________
Executed in   10.44 secs    fish           external
   usr time    9.91 secs    0.00 micros    9.91 secs
   sys time    0.35 secs  606.00 micros    0.35 secs

After

> du -sh community-general-3.7.0.tar.gz 
1.6M community-general-3.7.0.tar.gz

> du -sh community.general
9.1M community.general

> find . -type f | wc -l
738

> time ansible-galaxy collection install /home/dmsimard/dev/git/ansible_collections/community/general/community-general-3.7.0.tar.gz
Starting galaxy collection install process
Process install dependency map
Starting collection install process
Installing 'community.general:3.7.0' to '/home/dmsimard/.ansible/collections/ansible_collections/community/general'
community.general:3.7.0 was installed successfully
________________________________________________________
Executed in    6.75 secs    fish           external
   usr time    6.43 secs    0.00 micros    6.43 secs
   sys time    0.20 secs  614.00 micros    0.20 secs

Files that belong to the git repository such as git, github and azure pipeline configuration should not be included in the published collection tarball. Excluding tests saves on what makes the bulk of the size of the collection, both in size and number of files. In aggregate, excluding these files results in a meaningful improvement which will speed up the installation of the collection and reduce the size on disk.

dmsimard · 2021-10-05T21:06:42Z

Marked as WIP while we discuss ansible-community/community-topics#29

felixfontein · 2021-10-06T03:31:38Z

galaxy.yml

+  - .azure-pipelines
+  - .github
+  - .gitignore
+  - changelogs


Suggested change

- changelogs

It is absolutely essential that changelogs/changelog.yaml is included in the build. Otherwise building the Ansible changelog will not work.

Can you expand on that ? I was under the impression that leaving CHANGELOG.rst was sufficient ?

changelogs/changelog.yaml is the source of truth, both for CHANGELOG.rst (included in the collection) and the Ansible changelog are generated from it. The .rst file is only for the convenience of users and maintainers as it's easier to read, and also can have any other name or location.

felixfontein

Adding a negative review to prevent accidental merging. There are some legal questions that need to be resolved first before this can be merged (next to the necessary change w.r.t. changelog).

gotmax23 · 2022-08-02T11:08:48Z

Has there been any progress on this? What are the legal issues here?

felixfontein · 2022-08-02T17:13:53Z

There is no progress (to my knowledge). The legal problem still is (if I remember correctly): the collection tarball acts both as the source and the installable artefact. Excluding files used for the development from it would violate the GPL.

bcoca · 2022-08-02T17:30:04Z

@felixfontein IANAL but even including a URI to those files should be sufficient, the files and any changes must be available, but it does not mean every possible form of distribution of the product must have them ... otherwise distributing binary packages would be untenable.

felixfontein · 2022-08-03T20:11:59Z

I'm not sure what the exact problems are, the community team has been in (non-?)communication with RH legal on this matter for ... quite a long time now. I don't know exactly what they asked, but so far there was no answer AFAIK.

gotmax23 · 2022-08-10T16:10:03Z

I've created ansible-community/community-topics#126 to discuss this.

dmsimard · 2022-08-11T20:33:39Z

I'm not sure what the exact problems are, the community team has been in (non-?)communication with RH legal on this matter for ... quite a long time now. I don't know exactly what they asked, but so far there was no answer AFAIK.

This is a fair assessment -- I do not remember when because it's been so long now and I'm no longer employed at Red Hat to check my emails but I opened a ticket with Red Hat open source legal describing this issue and asking for advice but have not heard back.

There is a ticket number somewhere if someone would like to follow up internally but I do not know what it is.
It is possible that @gundalow could be able to find it.

felixfontein · 2022-11-23T18:27:07Z

Looks like we can continue with this, according to ansible-community/community-topics#131. I would still keep the .md files, and changelogs definitely must stay.

felixfontein · 2022-11-23T18:28:56Z

Also this needs a changelog fragment. IMO this is a breaking change and has to wait for the next major release. (Someone could depend on importing from ansible_collections.community.general.tests.unit for example.)

gotmax23 · 2022-11-23T19:07:58Z

The list I use for the Fedora ansible-collection-community-general package is:

.azure-pipelines
.github
.gitignore
.pre-commit-config.yaml
changelogs/.gitignore
changelogs/fragments
tests

dmsimard · 2022-11-23T22:10:40Z

Feel free to open a new PR, close this one and link to it for context.

I've been out of the loop for long enough and don't mind.

webknjaz · 2022-11-24T18:22:05Z

Excluding tests saves on what makes the bulk of the size of the
collection, both in size and number of files.

Sounds like collections really need an sdist vs a wheel separation too...

gotmax23 · 2022-11-24T18:33:19Z

Excluding tests saves on what makes the bulk of the size of the
collection, both in size and number of files.

Sounds like collections really need an sdist vs a wheel separation too...

I strongly agree. The lack of collection source distributions presents a problem for the ansible package, because these files are removed from the ansible sdist if they are removed from the Galaxy artifact.

felixfontein · 2022-11-24T19:28:43Z

I also strongly agree (and I think I mentioned more than once during community meetings that this would be really helpful and solve most of our problems :) ).

gotmax23 · 2022-11-24T21:21:41Z

Sounds like collections really need an sdist vs a wheel separation too...

How can we move this forward? Should I create a community-topic? I guess we'd need agreement from the core team and the Galaxy team. IIRC, @jctanner mentioned that this would be difficult to implement with Galaxy NG's current architecture when we last discussed this.

felixfontein · 2022-11-25T20:04:50Z

@gotmax23 a community topic would be great. Another possible place would be https://github.com/ansible/proposals/, but since there should be something in https://github.com/ansible-community/community-topics/, even if just to coordinate everything, I guess we should start with that.

gotmax23 · 2022-11-25T22:03:28Z

I've opened ansible-community/community-topics#161

dmsimard marked this pull request as draft October 5, 2021 20:56

ansibullbot added WIP Work in progress bug This issue/PR relates to a bug labels Oct 5, 2021

dmsimard mentioned this pull request Oct 5, 2021

Smaller ansible packages (no tests/ dir?) ansible-community/community-topics#29

Closed

ansibullbot added the has_issue label Oct 5, 2021

felixfontein reviewed Oct 6, 2021

View reviewed changes

felixfontein requested changes Oct 9, 2021

View reviewed changes

ansibullbot added the stale_ci CI is older than 7 days, rerun before merging label Oct 17, 2021

felixfontein mentioned this pull request Nov 24, 2021

Community Working Group Meeting Agenda ansible/community#539

Closed

felixfontein mentioned this pull request Apr 15, 2022

Remove unnecessary files from built collection tarball #3963

Closed

gotmax23 mentioned this pull request Aug 10, 2022

Discuss removing unnecessary files from collections ansible-community/community-topics#126

Closed

felixfontein added breaking_change This PR contains a breaking change that MUST NOT be backported check-before-release PR will be looked at again shortly before release and merged if possible. labels Nov 23, 2022

ansibullbot added needs_ci This PR requires CI testing to be performed. Please close and re-open this PR to trigger CI and removed stale_ci CI is older than 7 days, rerun before merging labels Jul 24, 2023

ansibullbot added the needs_rebase https://docs.ansible.com/ansible/devel/dev_guide/developing_rebasing.html label Nov 29, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

galaxy.yml: exclude non-collection files from collection build #3517

galaxy.yml: exclude non-collection files from collection build #3517

dmsimard commented Oct 5, 2021

dmsimard commented Oct 5, 2021

felixfontein Oct 6, 2021

dmsimard Oct 6, 2021

felixfontein Oct 6, 2021

felixfontein left a comment

gotmax23 commented Aug 2, 2022

felixfontein commented Aug 2, 2022

bcoca commented Aug 2, 2022

felixfontein commented Aug 3, 2022

gotmax23 commented Aug 10, 2022

dmsimard commented Aug 11, 2022

felixfontein commented Nov 23, 2022

felixfontein commented Nov 23, 2022

gotmax23 commented Nov 23, 2022

dmsimard commented Nov 23, 2022

webknjaz commented Nov 24, 2022

gotmax23 commented Nov 24, 2022

felixfontein commented Nov 24, 2022

gotmax23 commented Nov 24, 2022

felixfontein commented Nov 25, 2022

gotmax23 commented Nov 25, 2022

galaxy.yml: exclude non-collection files from collection build #3517

Are you sure you want to change the base?

galaxy.yml: exclude non-collection files from collection build #3517

Conversation

dmsimard commented Oct 5, 2021

SUMMARY

ISSUE TYPE

COMPONENT NAME

ADDITIONAL INFORMATION

Before

After

dmsimard commented Oct 5, 2021

felixfontein Oct 6, 2021

Choose a reason for hiding this comment

dmsimard Oct 6, 2021

Choose a reason for hiding this comment

felixfontein Oct 6, 2021

Choose a reason for hiding this comment

felixfontein left a comment

Choose a reason for hiding this comment

gotmax23 commented Aug 2, 2022

felixfontein commented Aug 2, 2022

bcoca commented Aug 2, 2022

felixfontein commented Aug 3, 2022

gotmax23 commented Aug 10, 2022

dmsimard commented Aug 11, 2022

felixfontein commented Nov 23, 2022

felixfontein commented Nov 23, 2022

gotmax23 commented Nov 23, 2022

dmsimard commented Nov 23, 2022

webknjaz commented Nov 24, 2022

gotmax23 commented Nov 24, 2022

felixfontein commented Nov 24, 2022

gotmax23 commented Nov 24, 2022

felixfontein commented Nov 25, 2022

gotmax23 commented Nov 25, 2022