[BEAM-8512] Add integration tests for flink_runner.py #9998
Conversation
|
CC @tweise |
|
Run Seed Job |
|
Run Python 2 PostCommit |
|
Run Python 3.7 PostCommit |
| @@ -1953,8 +1953,10 @@ class BeamModulePlugin implements Plugin<Project> { | |||
| } | |||
| project.ext.addPortableWordCountTasks = { | |||
| -> | |||
| addPortableWordCountTask(false) | |||
| addPortableWordCountTask(true) | |||
| addPortableWordCountTask(false, "PortableRunner") | |||
There was a problem hiding this comment.
Is there anything that we are testing with PortableRunner that isn't included in FlinkRunner? If not, perhaps just replace those?
Side question for follow-up: Since these tasks are Flink specific, should they be renamed and possibly moved out of BeamModulePlugin?
There was a problem hiding this comment.
Testing PortableRunner independently is important. Theoretically, the PortableRunner is a subset of FlinkRunner, but there are still things that can go wrong with the PortableRunner when it runs without the orchestration support of the FlinkRunner.
There was a problem hiding this comment.
Theoretically, the PortableRunner is a subset of FlinkRunner, but there are still things that can go wrong
+1 As long as PortableRunner remains a publicly documented option (and it will probably have to for at least a little while longer) we should test it. Also, they are testing slightly different things -- PortableRunner starts a containerized job server, while FlinkRunner (when local) starts the job server in a Java subprocess.
Since these tasks are Flink specific, should they be renamed and possibly moved out of BeamModulePlugin?
I'm going to add these tasks to Spark (eventually).
ibzib
force-pushed
the
flink-runner-test
branch
from
d905f8e
to
9322a29
Compare
|
Run Python 3.7 PostCommit |
ibzib
force-pushed
the
flink-runner-test
branch
from
9322a29
to
d692d2f
Compare
|
Run Portable_Python PreCommit |
|
Run Python 3.7 PostCommit |
1 similar comment
|
Run Python 3.7 PostCommit |
This is just a very basic smoke test to ensure that FlinkRunner works as intended with the default settings. This does not test behavior with an actual Flink cluster.
Post-Commit Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.