Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EQL: [Tests] enable server side debugging #64308

Merged
merged 4 commits into from
Oct 30, 2020

Conversation

matriv
Copy link
Contributor

@matriv matriv commented Oct 28, 2020

Register a new task runEqlCorrectnessNode which enables developers to
start an ES node in debug mode, properly restore the correctness data
and then run queries against it.

Register a new task `runEqlCorrectnessNode` which enables developers to
start an ES node in debug mode, properly restore the correctness data
and then run queries against it.
@matriv matriv added >test Issues or PRs that are addressing/adding tests v8.0.0 :Analytics/EQL EQL querying v7.11.0 v7.10.1 labels Oct 28, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-ql (:Query Languages/EQL)

@elasticmachine elasticmachine added the Team:QL (Deprecated) Meta label for query languages team label Oct 28, 2020
Copy link
Contributor

@astefan astefan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some comments.
Thank you for adding this task, for me at least is really helpful.

@@ -105,18 +77,6 @@ public static void logTotalExecutionTime() {
LOGGER.info("Total time: {} ms", totalTime);
}

@AfterClass
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why removing this method?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because when you have a running ES node and you try to execute a single query against it with gradle, the termination of the suite will delete the data, and you'd have to rerun the EqlDataLoader.

// Need to setup the log configuration properly to avoid messages when creating a new RestClient
PluginManager.addPackage(LogConfigurator.class.getPackage().getName());

Properties configuration = loadConfiguration();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could load the Properties inside the try() { .... } block.

}

static Properties loadConfiguration() throws IOException {
try (InputStream is = EsEQLCorrectnessIT.class.getClassLoader().getResourceAsStream(PROPERTIES_FILENAME)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be 100% correct, I think it should be EqlDataLoader's class loader to load the properties file.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, copy paste leftover.

Once the ES node is up and running, the data can be restored from the snapshot by running the `main` of the
`EqlDataLoader` class.

Once the data is loaded, a specific query can ber run against the running ES node with:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo: can ber run

If one wants to run an ES node manually (most probably to be able to debug the server), needs to run the following:

```shell script
./gradlew -p x-pack/plugin/eql/qa/correctness runEqlCorrectnessNode --debug-jvm
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If below you provided an example gradlew command without using -p, I think it's better to be consistent and either use here the same format as the other command or use -p in both places. Not sure of the differences though, but I've been advised lately to stop using the -p format in favor of something like gradlew :x-pack:plugin:eql:qa:correctness:runEqlCorrectness --debug-jvm

RequestOptions.DEFAULT
);
}
EqlDataLoader.restoreSnapshot(client(), highLevelClient(), CFG);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The highLevelClient has a reference to the rest client already (through getLowLevelClient()), I don't think you need to pass client() here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thx, missed that.

Copy link
Contributor

@astefan astefan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@@ -5,10 +5,11 @@
#

index_name=mitre
index_doc_count=3950632
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@@ -36,3 +44,8 @@ tasks.named('javaRestTest').configure {
showStandardStreams = true
}
}

tasks.register("runEqlCorrectnessNode", RunTask) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 LGTM

@matriv matriv merged commit fc8c6dd into elastic:master Oct 30, 2020
@matriv matriv deleted the debug-eql-correctness branch October 30, 2020 21:26
matriv added a commit to matriv/elasticsearch that referenced this pull request Oct 30, 2020
Register a new task `runEqlCorrectnessNode` which enables developers to
start an ES node in debug mode, properly restore the correctness data
and then run queries against it.

Assert the index is restored correctly and use new snapshot.

(cherry picked from commit fc8c6dd)
matriv added a commit to matriv/elasticsearch that referenced this pull request Oct 30, 2020
Register a new task `runEqlCorrectnessNode` which enables developers to
start an ES node in debug mode, properly restore the correctness data
and then run queries against it.

Assert the index is restored correctly and use new snapshot.

(cherry picked from commit fc8c6dd)
matriv added a commit that referenced this pull request Oct 31, 2020
Register a new task `runEqlCorrectnessNode` which enables developers to
start an ES node in debug mode, properly restore the correctness data
and then run queries against it.

Assert the index is restored correctly and use new snapshot.

(cherry picked from commit fc8c6dd)
matriv added a commit that referenced this pull request Oct 31, 2020
Register a new task `runEqlCorrectnessNode` which enables developers to
start an ES node in debug mode, properly restore the correctness data
and then run queries against it.

Assert the index is restored correctly and use new snapshot.

(cherry picked from commit fc8c6dd)
@andreidan andreidan added v7.10.0 and removed v7.10.1 labels Nov 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Analytics/EQL EQL querying Team:QL (Deprecated) Meta label for query languages team >test Issues or PRs that are addressing/adding tests v7.10.0 v7.11.0 v8.0.0-alpha1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants