Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Elastic adapter #2727

Merged
merged 33 commits into from
May 24, 2024
Merged
Changes from 1 commit
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
998c893
Add stub source and elastic API funcs
Apr 12, 2024
b9a7c9a
Spawn workers and ship chunks
Apr 17, 2024
e6ea20d
Now successfully detects a credential
Apr 21, 2024
4624cef
Linting fixes
Apr 22, 2024
c6bb4b8
Add integration tests and resolve some bugs they uncovered
Apr 27, 2024
ca89bc2
Logstash -> Elasticsearch
Apr 27, 2024
8b1dda6
Add support for --index-pattern
camgunz Apr 27, 2024
dc4397e
Add support for --query-json
camgunz Apr 27, 2024
b7bd504
Use structs instead of string building to construct a search body
camgunz Apr 28, 2024
35748c9
Support --since-timestamp
camgunz Apr 28, 2024
ae2cfe5
Implement additional authentication methods
camgunz Apr 28, 2024
a8dda1d
Fix some small bugs
camgunz Apr 29, 2024
f862ac4
Refactoring to support --best-effort-scan
camgunz May 3, 2024
ef99cf9
Finish implementation of --best-effort-scan
camgunz May 4, 2024
96f2c44
Implement scan catch-up
camgunz May 5, 2024
7564385
Finish connecting support for nodes CLI arg
camgunz May 5, 2024
ced2a76
Add some integration tests around the catchup mechanism
camgunz May 8, 2024
131d0a7
Merge branch 'main' into elastic-adapter
camgunz May 10, 2024
0eda2ac
go mod tidy
camgunz May 10, 2024
826fc8b
Fix some linting issues
camgunz May 15, 2024
d566f2a
Merge remote-tracking branch 'upstream/main' into elastic-adapter
camgunz May 15, 2024
4b3e259
Remove some debugging Prints
camgunz May 15, 2024
27b274e
Move off of _doc
camgunz May 17, 2024
c53abb9
Remove informational Printf and add informational logging
camgunz May 24, 2024
c67fdf1
Remove debugging logging
camgunz May 24, 2024
e82a955
Copy the index from the outer loop as well
camgunz May 24, 2024
999a0f7
Don't burn up the ES API with rapid requests if there's no work to do…
camgunz May 24, 2024
f01404f
No need to export UnitOfWork.AddSearch
camgunz May 24, 2024
bca1376
Use a better name for the range query variable when building the time…
camgunz May 24, 2024
664048b
Replace some unlocking defers with explicit unlocks to make the synch…
camgunz May 24, 2024
a4457ee
found -> ok
camgunz May 24, 2024
bf9c93f
Remove superfluous buildElasticClient method
camgunz May 24, 2024
e7a8c2d
Merge remote-tracking branch 'upstream/main' into elastic-adapter
camgunz May 24, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
found -> ok
  • Loading branch information
camgunz committed May 24, 2024
commit a4457eef66a0fb03c26cf68cf3edf5fc876dc93b
12 changes: 6 additions & 6 deletions pkg/sources/elasticsearch/api.go
Original file line number Diff line number Diff line change
Expand Up @@ -247,8 +247,8 @@ func createPITForSearch(
return "", err
}

pitID, found := data["id"].(string)
if !found {
pitID, ok := data["id"].(string)
if !ok {
return "", errors.New("No id in response")
}

Expand Down Expand Up @@ -343,8 +343,8 @@ func processSearchedDocuments(
continue
}

jsonSort, found := hit["sort"].([]any)
if !found {
jsonSort, ok := hit["sort"].([]any)
if !ok {
continue
}

Expand Down Expand Up @@ -432,8 +432,8 @@ func (indices *Indices) Update(

newIndicesByName := make(map[string]*Index)
for _, name := range indexNames {
index, found := indicesByName[name]
if found {
index, ok := indicesByName[name]
if ok {
newIndicesByName[name] = index
} else {
index = NewIndex()
Expand Down