Adopt Conventional Commits

[assignUser]

With the Velox API maturing more and more there will be a point when releases become viable. At that point Velox will need a versioning schema, probably https://semver.org/ ?

Keeping track of the correct version for the next release manually can be complicated and prone to errors (arrow only releases major versions i.a. for this reason). A solution for this would be commit messages that contains a machine readable component that allows for a script to collect and summarize the changes and determine the correct version, e.g. https://www.conventionalcommits.org/en/v1.0.0/

@kgpai @raulcd @pedroerp what do you think?

[pedroerp]

That's an interesting idea. I wonder why arrow doesn't it? Have you guys tried it before?

[assignUser]

Mostly due to the strong interdependence of the C++ component libarrow with other components like Python and R. As the more unstable (api wise) acero engine is part of libarrow we currently stick with major releases. There is an ongoing discussion (e.g. here) within arrow about detailed versioning for the different components in the monorepo and efforts to make this easier like extracting the pyarrow C++ code from libarrow. We are slowly making progress towards something tangible but have not reached a conclusion yet.

[pedroerp]

We had an initial quick discussion about this and the sentiment was that it seemed a bit early for this at this point.

@mbasmanova @majetideepak @Yuhta what are your thoughts on this?

[assignUser]

Yes while the api is still not fixed in any reasonable way and everything is changing fast it does not really matter but I wanted to bring this up early enough so it can influence the longterm plan for the release process and make it as smooth and easy as possible. Both for the maintainers as well as the users.

For example for arrow we do major only releases to avoid having to track braking api changes, even if most of the apis are quite stable, as a major version bump in semver means breaking changes. This makes it easier but also has downsides for some components like pyarrow and arrow go where usually major versions of packages are pinned for compatibility. Releasing only major version breaks this custom and can lead to dependency hell and a lot more maintenance required for downstream users.

[mbasmanova]

I'm trying to understand this proposal. I imagine we would need to setup some sort of a release process where we freeze the repo, run some tests, fix bugs if any, then make a commit to bump up a version in some file. I'm not sure this can be easily automated. Am I missing something?

[assignUser]

Instead of a hard freeze were nothing is committed to main my suggestions would be to create a release branch. That release branch is "frozen" at the time of branching but work can continue on main. Any necessary bugfixes that would block the release can then be cherry picked from main to the release branch. This is how we do it in arrow.

But this would be still a bit manual, there are ways to just fully automate releases if the commit messages are used correctly e.g. (I have not used this specifically) https://github.com/semantic-release/semantic-release

Doing a release on every commit with code changes as the above tool recommends seems to be a bit excessive tbh, my proposal was mostly about the way conventional commits make it easier to determin the correct new version without manually checking/keeping track of it.

[mbasmanova]

That release branch is "frozen" at the time of branching but work can continue on main.

We have tried both approaches with Presto and I feel that releasing from main is a better solution. The challenge with a separate release branch is that a small set of folks (1-2) is trying to stabilize that branch without any help from the rest of the community. Folks continue landing lots of changes to main making it difficult to stabilize the release branch. I feel release go easier if everybody in the community is invested in making then happen quickly.

[assignUser]

that a small set of folks (1-2) is trying to stabilize that branch

hm yeah, for arrow we have a dedicated release manager according to ASF procedure so that is kind of intended. Ideally main should always be in a releaseable state but even with arrows expansive CI machinery that is not 100% guranted....

Folks continue landing lots of changes to main making it difficult to stabilize the release branch.

Why would changes to main influence stabilizing the release branch? The whole purpose of the release branch is to allow for work to continue unhindered on main while possibly needed changes get cherry picked but otherwise the branch is not updated?

[mbasmanova]

Ideally main should always be in a releaseable state

This is pretty hard to achieve in practice. Often, release requires shadowing a substantial subset of production workload which takes a long time and often discovers bugs.

Why would changes to main influence stabilizing the release branch?

Primarily because the release manager needs help from the community in resolving bugs / issues found in release branch and it has been difficult to get proper attention from the folks. It is also often becomes different to quickly fix a bug if more code has been added on top of the relevant code during the release.

[assignUser]

This is pretty hard to achieve in practice.

💯

it has been difficult to get proper attention from the folks.

I see, with arrow there was strong community opposition to releasing from the default branch but I see how doing a real freeze might focus overall community efforts into fixing the release. I am definitely open to giving that model a try :)

[tdcmeehan]

I wanted to see if we had any more thoughts on how to provide specific versions of Velox to users? Perhaps, in an effort to preserve the trunk based development model that the Velox community has adopted, main is frozen for an ideally short period while the release is finalized, and the final artifact is a tag and/or libraries folks can download to provide a specific version of Velox? In this model, we'd just provide major-only releases similar to Arrow. An open question is how to provide patch fixes for known bugs to previously releases versions.

[mbasmanova]

At this point, we don't seem to have resources to support more than one version of the library (which is trunk). It is also not clear how to verify a particular version and mark it as "stable" or similar. Any suggestions?

[tdcmeehan]

Perhaps a first step could be to provide a 0.x.y release system? Such releases are considered unstable by nature (c.f. https://semver.org/#spec-item-4), so it could help with the more narrow problem of how to provide a version of the library for consumers who need it and are comfortable depending on an initial development version.

[mbasmanova]

That's a sensible proposal, but also requires funding that's currently not available. BTW, who would be primarily beneficiaries of that today?

[tdcmeehan]

Understood on the current funding. Eventually, as Prestissimo adoption grows, I expect this would be needed as most external users would be using versions of Presto, and while Presto doesn't adopt semantic versioning, it does patch bugs when it's feasible (especially for correctness issues). I also wonder if this might help with Velox adoption in more systems, as it seems necessary to version a library to assist the user of the library to understand what has changed between points in time and when an upgrade is needed or desired. This might be premature--I'm simply thinking ahead and wondering where we are currently with respect to this eventual goal.

[pedroerp]

@tdcmeehan is this something that you or someone from your team would be willing to take as a project?

[assignUser]

Closed in favor of #11039