Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Smoke tests and samples #1193

Open
dlion opened this issue Aug 30, 2023 · 2 comments
Open

Smoke tests and samples #1193

dlion opened this issue Aug 30, 2023 · 2 comments

Comments

@dlion
Copy link
Member

dlion commented Aug 30, 2023

Summary

I noticed that some of our samples don't work properly with the current version of lifecycle. (i.e. buildpacks/samples#167, buildpacks/samples#163)

Those samples can be very useful for identifying possible regressions and for the community, we use them in our documentation, we often use them during our troubleshooting and we should treat them as a first-class citizens.

Manual solution

In order to verify that our code didn't have any regression and to maintain them up to date being sure that they work every time we should try them every new release of a stable version of pack/lifecycle.

Downside

It requires lots of time and effort from our side and we can potentially forget about it.


Proposal

So I thought about an automatic and more efficient way to do so:

My proposal would be to have some smoke tests that we run (or that our CI runs -better-) every time we release a new version of lifecycle/pack towards those examples.
It will force us to update those samples with the latest changes in order to release a stable version and of course to verify that we didn't introduce any regression.

The process can looks like:

  1. Build lifecycle
  2. Run lifecycle components one by one using our samples
  3. Do that for every important/basic scenario
  4. Check for failures
  5. If every thing went well we can release otherwise we need to adjust our samples first.

Context

https://www.guru99.com/smoke-testing.html
https://martinfowler.com/articles/practical-test-pyramid.html

@dlion dlion changed the title Smoke tests Smoke tests and samples Aug 30, 2023
@natalieparellano
Copy link
Member

Related: #801

@natalieparellano
Copy link
Member

Blocking on other issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants