Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Elastic adapter #2727

Merged
merged 33 commits into from
May 24, 2024
Merged

Elastic adapter #2727

merged 33 commits into from
May 24, 2024

Conversation

camgunz
Copy link
Contributor

@camgunz camgunz commented Apr 22, 2024

Description:

This PR is to add an elasticsearch source.

Checklist:

  • [*] Tests passing (make test-community)?
    • Looks like some issues w/ the tar source?
  • [*] Lint passing (make lint this requires golangci-lint)?
    • unrelated issues in LDAP seems like

@camgunz camgunz requested review from a team as code owners April 22, 2024 16:00
@CLAassistant
Copy link

CLAassistant commented Apr 22, 2024

CLA assistant check
All committers have signed the CLA.

unitsOfWork := distributeDocumentScans(&indices, s.concurrency, scanCoverageRate)

for uowIndex, outerUOW := range unitsOfWork {
uow := outerUOW
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@camgunz
Suggestion: Even though uowIndex is only used in logs, it still needs to have a copy similar to uow inside the loop otherwise we have the potential for incorrect log output

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oof, nice catch; I think this started life as _ but then I used it for logging and missed pulling it in 👍

}
}

func (uow *UnitOfWork) AddSearch(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: any reason we need to export this function? Looks like the only place it's being used is distributeDocumentScans and the tests.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah you're right, no need to export here 👍


func (i *Index) DocumentAlreadySeen(document *Document) bool {
i.lock.Lock()
defer i.lock.Unlock()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: rather than locking at the top and deferring the unlock we should just surround contentious access with straight up lock/unlocks rather than defer. Easier to figure out what needs to be protected that way.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting, yeah I buy that. I'll change what I can -- in this case I'll probably keep defer and just move it below ParseTimestamp because everything else needs to be write synchronized, but I think the others can all be explicit.

return "", err
}

pitID, found := data["id"].(string)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use ok instead of found for consistency

@zricethezav zricethezav merged commit 311494e into trufflesecurity:main May 24, 2024
11 of 12 checks passed
haraldh referenced this pull request in matter-labs/vault-auth-tee Jun 10, 2024
[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
|
[trufflesecurity/trufflehog](https://togithub.com/trufflesecurity/trufflehog)
| action | minor | `v3.76.3` -> `v3.78.0` |

---

### Release Notes

<details>
<summary>trufflesecurity/trufflehog
(trufflesecurity/trufflehog)</summary>

###
[`v3.78.0`](https://togithub.com/trufflesecurity/trufflehog/releases/tag/v3.78.0)

[Compare
Source](https://togithub.com/trufflesecurity/trufflehog/compare/v3.77.0...v3.78.0)

#### What's Changed

- Add postman to tui by [@&#8203;hxnyk](https://togithub.com/hxnyk) in
[https://github.com/trufflesecurity/trufflehog/pull/2895](https://togithub.com/trufflesecurity/trufflehog/pull/2895)
- chore(deps): update alpine docker tag to v3.20 by
[@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2874](https://togithub.com/trufflesecurity/trufflehog/pull/2874)
- fix(deps): update golang.org/x/exp digest to
[`fd00a4e`](https://togithub.com/trufflesecurity/trufflehog/commit/fd00a4e)
by [@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2899](https://togithub.com/trufflesecurity/trufflehog/pull/2899)
- Update metadata for DataDog for API + APPKey by
[@&#8203;abmussani](https://togithub.com/abmussani) in
[https://github.com/trufflesecurity/trufflehog/pull/2879](https://togithub.com/trufflesecurity/trufflehog/pull/2879)
- Consistent docker image of MSSQL for integration testing. by
[@&#8203;abmussani](https://togithub.com/abmussani) in
[https://github.com/trufflesecurity/trufflehog/pull/2898](https://togithub.com/trufflesecurity/trufflehog/pull/2898)
- fix(deps): update module github.com/charmbracelet/bubbletea to v0.26.4
by [@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2885](https://togithub.com/trufflesecurity/trufflehog/pull/2885)
- Remove 'www' from `DefaultFalsePositives` by
[@&#8203;rgmz](https://togithub.com/rgmz) in
[https://github.com/trufflesecurity/trufflehog/pull/2896](https://togithub.com/trufflesecurity/trufflehog/pull/2896)
- Redis integration test by
[@&#8203;abmussani](https://togithub.com/abmussani) in
[https://github.com/trufflesecurity/trufflehog/pull/2901](https://togithub.com/trufflesecurity/trufflehog/pull/2901)
- Fix Github `enumerateWithToken` test failure by
[@&#8203;rgmz](https://togithub.com/rgmz) in
[https://github.com/trufflesecurity/trufflehog/pull/2880](https://togithub.com/trufflesecurity/trufflehog/pull/2880)
- fix(deps): update module github.com/aws/aws-sdk-go to v1.53.14 by
[@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2900](https://togithub.com/trufflesecurity/trufflehog/pull/2900)
- fix(deps): update module github.com/hashicorp/go-retryablehttp to
v0.7.7 by [@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2904](https://togithub.com/trufflesecurity/trufflehog/pull/2904)
- integration testing for mongodb. by
[@&#8203;abmussani](https://togithub.com/abmussani) in
[https://github.com/trufflesecurity/trufflehog/pull/2907](https://togithub.com/trufflesecurity/trufflehog/pull/2907)
- fix(deps): update module
github.com/azure/go-autorest/autorest/azure/auth to v0.5.13 by
[@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2902](https://togithub.com/trufflesecurity/trufflehog/pull/2902)
- chore: fix some comments by
[@&#8203;jinjiadu](https://togithub.com/jinjiadu) in
[https://github.com/trufflesecurity/trufflehog/pull/2903](https://togithub.com/trufflesecurity/trufflehog/pull/2903)
- \[chore] Always log git repositories being scanned by
[@&#8203;mcastorina](https://togithub.com/mcastorina) in
[https://github.com/trufflesecurity/trufflehog/pull/2909](https://togithub.com/trufflesecurity/trufflehog/pull/2909)
- Add Jenkins scanning by
[@&#8203;dustin-decker](https://togithub.com/dustin-decker) in
[https://github.com/trufflesecurity/trufflehog/pull/2892](https://togithub.com/trufflesecurity/trufflehog/pull/2892)

#### New Contributors

- [@&#8203;jinjiadu](https://togithub.com/jinjiadu) made their first
contribution in
[https://github.com/trufflesecurity/trufflehog/pull/2903](https://togithub.com/trufflesecurity/trufflehog/pull/2903)

**Full Changelog**:
trufflesecurity/trufflehog@v3.77.0...v3.78.0

###
[`v3.77.0`](https://togithub.com/trufflesecurity/trufflehog/releases/tag/v3.77.0)

[Compare
Source](https://togithub.com/trufflesecurity/trufflehog/compare/v3.76.3...v3.77.0)

#### What's Changed

- Remove "finished verificationOverlap chunks" log line by
[@&#8203;rgmz](https://togithub.com/rgmz) in
[https://github.com/trufflesecurity/trufflehog/pull/2860](https://togithub.com/trufflesecurity/trufflehog/pull/2860)
- fix(deps): update module github.com/wasilibs/go-re2 to v1.5.3 by
[@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2861](https://togithub.com/trufflesecurity/trufflehog/pull/2861)
- fix(deps): update module google.golang.org/api to v0.181.0 by
[@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2857](https://togithub.com/trufflesecurity/trufflehog/pull/2857)
- fix(deps): update module github.com/aws/aws-sdk-go to v1.53.5 by
[@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2859](https://togithub.com/trufflesecurity/trufflehog/pull/2859)
- Update azure storage extra data by
[@&#8203;abmussani](https://togithub.com/abmussani) in
[https://github.com/trufflesecurity/trufflehog/pull/2808](https://togithub.com/trufflesecurity/trufflehog/pull/2808)
- Update regex for Organization in Azure Devops detector by
[@&#8203;abmussani](https://togithub.com/abmussani) in
[https://github.com/trufflesecurity/trufflehog/pull/2866](https://togithub.com/trufflesecurity/trufflehog/pull/2866)
- fix(deps): update module github.com/aws/aws-sdk-go to v1.53.6 by
[@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2867](https://togithub.com/trufflesecurity/trufflehog/pull/2867)
- \[chore] - Use http.NewRequestWithContext by
[@&#8203;ahrav](https://togithub.com/ahrav) in
[https://github.com/trufflesecurity/trufflehog/pull/2870](https://togithub.com/trufflesecurity/trufflehog/pull/2870)
- adding Groq detector by [@&#8203;0x1](https://togithub.com/0x1) in
[https://github.com/trufflesecurity/trufflehog/pull/2873](https://togithub.com/trufflesecurity/trufflehog/pull/2873)
- Log reasons for GitLab repo exclusion by
[@&#8203;rosecodym](https://togithub.com/rosecodym) in
[https://github.com/trufflesecurity/trufflehog/pull/2875](https://togithub.com/trufflesecurity/trufflehog/pull/2875)
- \[github] Scan user repositories by
[@&#8203;rgmz](https://togithub.com/rgmz) in
[https://github.com/trufflesecurity/trufflehog/pull/2814](https://togithub.com/trufflesecurity/trufflehog/pull/2814)
- Elastic adapter by [@&#8203;camgunz](https://togithub.com/camgunz) in
[https://github.com/trufflesecurity/trufflehog/pull/2727](https://togithub.com/trufflesecurity/trufflehog/pull/2727)
- Improve handling of Gist URLs by
[@&#8203;rgmz](https://togithub.com/rgmz) in
[https://github.com/trufflesecurity/trufflehog/pull/2653](https://togithub.com/trufflesecurity/trufflehog/pull/2653)
- Fix some GitHub source test issues by
[@&#8203;rgmz](https://togithub.com/rgmz) in
[https://github.com/trufflesecurity/trufflehog/pull/2774](https://togithub.com/trufflesecurity/trufflehog/pull/2774)
- fix(deps): update module github.com/aws/aws-sdk-go to v1.53.10 by
[@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2871](https://togithub.com/trufflesecurity/trufflehog/pull/2871)
- fix(deps): update module github.com/go-logr/logr to v1.4.2 by
[@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2869](https://togithub.com/trufflesecurity/trufflehog/pull/2869)
- fix(deps): update module cloud.google.com/go/secretmanager to v1.13.1
by [@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2884](https://togithub.com/trufflesecurity/trufflehog/pull/2884)
- fix(deps): update golang.org/x/exp digest to
[`4c93da0`](https://togithub.com/trufflesecurity/trufflehog/commit/4c93da0)
by [@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2883](https://togithub.com/trufflesecurity/trufflehog/pull/2883)
- fix(deps): update module github.com/elastic/go-elasticsearch/v8 to
v8.13.1 by [@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2886](https://togithub.com/trufflesecurity/trufflehog/pull/2886)
- fix(deps): update module github.com/gabriel-vasile/mimetype to v1.4.4
by [@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2890](https://togithub.com/trufflesecurity/trufflehog/pull/2890)
- Added extra data for LaunchDarkly by
[@&#8203;abmussani](https://togithub.com/abmussani) in
[https://github.com/trufflesecurity/trufflehog/pull/2836](https://togithub.com/trufflesecurity/trufflehog/pull/2836)
- feat: support docker image history scanning by
[@&#8203;jamestelfer](https://togithub.com/jamestelfer) in
[https://github.com/trufflesecurity/trufflehog/pull/2882](https://togithub.com/trufflesecurity/trufflehog/pull/2882)

#### New Contributors

- [@&#8203;camgunz](https://togithub.com/camgunz) made their first
contribution in
[https://github.com/trufflesecurity/trufflehog/pull/2727](https://togithub.com/trufflesecurity/trufflehog/pull/2727)
- [@&#8203;jamestelfer](https://togithub.com/jamestelfer) made their
first contribution in
[https://github.com/trufflesecurity/trufflehog/pull/2882](https://togithub.com/trufflesecurity/trufflehog/pull/2882)

**Full Changelog**:
trufflesecurity/trufflehog@v3.76.3...v3.77.0

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/matter-labs/vault-auth-tee).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zNzcuOCIsInVwZGF0ZWRJblZlciI6IjM3LjM4OC4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

3 participants