Add support for adhoc fuzzer job #7810
Conversation
✅ Deploy Preview for meta-velox canceled.
|
facebook-github-bot
added
the
CLA Signed
kgpai
requested review from
bikramSingh91,
pedroerp,
laithsakka and
assignUser
and removed request for
pedroerp
kgpai
force-pushed
the
add_adhoc_fuzzer_job
branch
from
d32fe88
to
9b4138a
Compare
|
@kgpai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
| branchRef: | ||
| description: 'Branch to checkout out' | ||
| default: 'main' |
There was a problem hiding this comment.
I would rename to ref as the checkout action takes branches or refs for its ref: input.
| branchRef: | |
| description: 'Branch to checkout out' | |
| default: 'main' | |
| ref: | |
| description: 'Branch or ref to checkout out' | |
| default: 'main' |
| submodules: 'recursive' | ||
|
|
There was a problem hiding this comment.
Same for the other checkouts to enable running on a different ref:
| submodules: 'recursive' | |
| ref: "${{ inputs.ref }}" | |
| submodules: 'recursive' | |
|
|
||
| - name: "Build" | ||
| run: | | ||
| make debug NUM_THREADS="${{ github.event.inputs.numThreads }}" MAX_HIGH_MEM_JOBS="${{ github.event.inputs.maxHighMemJobs }}" MAX_LINK_JOBS="${{ github.event.inputs.maxLinkJobs }}" EXTRA_CMAKE_FLAGS="${{ github.event.inputs.extraCMakeFlags }}" |
There was a problem hiding this comment.
You can use the inputs context directly. See https://docs.github.com/en/actions/learn-github-actions/contexts#inputs-context for details
| make debug NUM_THREADS="${{ github.event.inputs.numThreads }}" MAX_HIGH_MEM_JOBS="${{ github.event.inputs.maxHighMemJobs }}" MAX_LINK_JOBS="${{ github.event.inputs.maxLinkJobs }}" EXTRA_CMAKE_FLAGS="${{ github.event.inputs.extraCMakeFlags }}" | |
| make debug NUM_THREADS="${{ inputs.numThreads }}" MAX_HIGH_MEM_JOBS="${{ inputs.maxHighMemJobs }}" MAX_LINK_JOBS="${{ inputs.maxLinkJobs }}" EXTRA_CMAKE_FLAGS="${{ inputs.extraCMakeFlags }}" |
| submodules: 'recursive' | ||
|
|
||
| - name: "Install dependencies" | ||
| run: source ./scripts/setup-ubuntu.sh |
There was a problem hiding this comment.
You can also run this in the same docker container used in cci via container: but we can do that in a follow up as this job is not run regularly.
| description: 'Additional CMake flags' | ||
| default: '-DVELOX_ENABLE_ARROW=ON' | ||
|
|
||
|
|
There was a problem hiding this comment.
workflow_dispatch is privileged trigger so we want to limit permissions to the necessary minimum.
| permissions: | |
| contents: read | |
|
|
||
| name: "Adhoc Fuzzer Jobs" | ||
|
|
||
| on: |
There was a problem hiding this comment.
It would be good to also add pull_request with a path filter set to this workflow that changes to the workflow can be tested when submitting a PR but that will also require some additional defaults as there will be no inputs iirc.
|
Closed in favor of #7842 |
Summary: This PR tackles several issues with the scheduled fuzzer workflow: - workaround compilation being OOMkilled by using only 8 threads and using `mold` as a linker - this is a stopgap measure only, we should really look at our build times/footprint and see how we can optimize that as a 45 minute build on an 8 core machine is pretty massive. - only compile once (saves ~ 1.5h of CI time) - use actions/cache to save the ccache to further improve build times - add `workflow_dispatch` trigger to enable ad hoc runs of the fuzzer jobs - add `pull_request` trigger to enable testing of the workflow in PRs (like this one ^^) - we could easily move the remaining CCI fuzzer job over to this workflow as well Reverts #7811 to use 8-core machines again. Supersedes #7810 Pull Request resolved: #7842 Reviewed By: mbasmanova Differential Revision: D51960006 Pulled By: kgpai fbshipit-source-id: 0f493f80d5a87cca20bd53f1074a5f4622ee02b5
This PR will allow us to run adhoc fuzzer job against ,
1 . Any branch / sha
2. Configure the build process for threads/linkjobs/ Cmake flags for additional customizability.
The job can be accessed via the Github Actions UI.
This should be useful for us in cases, where we would like to test more fuzzer coverage whithout having to have the code checked in and waiting for the nightly run.