ci: Only run E2E test apps that are affected on PRs #14254
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
size-limit report 📦
|
mydea
force-pushed
the
fn/only-run-changed-e2e-tests
branch
from
ae3d63c to
fcd6b78
Compare
mydea
force-pushed
the
fn/only-run-changed-e2e-tests
branch
3 times, most recently
from
927af9e to
c7df3bf
Compare
mydea
requested review from
lforst,
Lms24,
AbhiPrasad,
chargome and
andreiborza
lforst
reviewed
Nov 14, 2024
lforst
approved these changes
Nov 14, 2024
chargome
approved these changes
Nov 14, 2024
AbhiPrasad
reviewed
Nov 14, 2024
Comment on lines
23
to
32
| const args: Record<string, string> = {}; | ||
| process.argv | ||
| .slice(2) | ||
| .filter(arg => arg.startsWith('--') && arg.includes('=')) | ||
| .forEach(arg => { | ||
| const [part1, part2] = arg.split('=') as [string, string]; | ||
| const argName = part1.replace('--', ''); | ||
| const argValue = part2; | ||
| args[argName] = argValue; | ||
| }); |
There was a problem hiding this comment.
m: We can pass process.argv into https://nodejs.org/api/util.html#utilparseargsconfig and use it instead.
There was a problem hiding this comment.
ah, I was looking for this (remembered that something like this exists, but was not sure anymore how/where) - that's much better!
There was a problem hiding this comment.
this also required bumping the node types in the e2e-tests package, but that should be OK i'd say (as we use node 18 anyhow).
| ...packageJson.dependencies, | ||
| }; | ||
|
|
||
| return Object.keys(dependencies).filter(key => key.startsWith('@sentry')); |
There was a problem hiding this comment.
should we also filter @sentry-internal?
There was a problem hiding this comment.
I feel like if some e2e test has a dependency on some internal package, we should also re-run it in this case? the classic thing would be @sentry-internal/test-utils, IMHO changes there should trigger all e2e tests that use this (which is all of them 😅 )
This reverts commit e91d64d.
mydea
force-pushed
the
fn/only-run-changed-e2e-tests
branch
from
2f61b7f to
e68dc71
Compare
onurtemizkan
pushed a commit
that referenced
this pull request
Nov 22, 2024
This PR aims to solve two things: 1. Do not require us to keep a list of test-applications to run on CI. This is prone to be forgotten, and then certain tests do not run on CI and we may not notice (this has happened more than once in the past 😬 ) 2. Do not run E2E test applications that are unrelated to anything that was changed. With this PR, instead of keeping a list of E2E test jobs we want to run manually in the build.yml file, this is now inferred (on CI) by a script based on the nx dependency graph and some config in the test apps package.json. This is how it works: 1. Pick all folders in test-applications, by default all of them should run. --> This will "fix" the problem we had sometimes that we forgot to add test apps to the build.yml so they don't actually run on CI :grimacing: 3. For each test app, look at it's dependencies (and devDependencies), and see if any of them has changed in the current PR (based on nx affected projects). So e.g. if you change something in browser, only E2E test apps that have any dependency that uses browser will run, so e.g. node-express will be skipped. 4. Additionally, you can configure variants in the test apps package.json - instead of defining this in the workflow file, it has to be defined in the test app itself now. 5. Finally, a test app can be marked as optional in the package.json, and can also have optional variants to run. 6. The E2E test builds the required & optional matrix on the fly based on these things. 7. In pushes (=on develop/release branches) we just run everything. 8. If anything inside of the e2e-tests package changed, run everything. --> This could be further optimized to only run changed test apps, but this was the easy solution for now. ## Example test runs * In a PR with no changes related to the E2E tests, nothing is run: https://github.com/getsentry/sentry-javascript/actions/runs/11838604965 * In a CI run for a push, all E2E tests are run: https://github.com/getsentry/sentry-javascript/actions/runs/11835834748 * In a PR with a change in e2e-tests itself, all E2E tests are run: https://github.com/getsentry/sentry-javascript/actions/runs/11836668035 * In a PR with a change in the node package, only related E2E tests are run: https://github.com/getsentry/sentry-javascript/actions/runs/11838274483
s1gr1d
added a commit
that referenced
this pull request
Dec 5, 2024
Updates readme based on this PR: #14254
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR aims to solve two things:
With this PR, instead of keeping a list of E2E test jobs we want to run manually in the build.yml file, this is now inferred (on CI) by a script based on the nx dependency graph and some config in the test apps package.json. This is how it works:
Example test runs