Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use nailgun for javac and import parser #12982

Merged
merged 7 commits into from
Sep 24, 2021
Merged

Use nailgun for javac and import parser #12982

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
38d7ec1
Hold an async Mutex while using a NailgunProcess, and implement finge…
stuhood Sep 12, 2021
02531bf
is_nailgunnable: bool -> use_nailgun: Digest
stuhood Sep 12, 2021
00dfe52
Move workdir creation out into the local runner, since the nailgun ru…
stuhood Sep 13, 2021
c336fd8
Move to invoking via a classname so that native nailgun support can i…
stuhood Sep 11, 2021
918eab6
Adjust javac and import parser to use nailgun.
stuhood Sep 13, 2021
04f18e0
Review feedback.
stuhood Sep 24, 2021
dc1660d
Fixup tests.
stuhood Sep 24, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions src/python/pants/engine/process.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ class Process:
output_directories: Tuple[str, ...]
timeout_seconds: int | float
jdk_home: str | None
is_nailgunnable: bool
use_nailgun: Digest
execution_slot_variable: str | None
cache_scope: ProcessCacheScope

Expand All @@ -84,7 +84,7 @@ def __init__(
output_directories: Iterable[str] | None = None,
timeout_seconds: int | float | None = None,
jdk_home: str | None = None,
is_nailgunnable: bool = False,
use_nailgun: Digest = EMPTY_DIGEST,
execution_slot_variable: str | None = None,
cache_scope: ProcessCacheScope = ProcessCacheScope.SUCCESSFUL,
) -> None:
Expand Down Expand Up @@ -128,7 +128,7 @@ def __init__(
# NB: A negative or None time value is normalized to -1 to ease the transfer to Rust.
self.timeout_seconds = timeout_seconds if timeout_seconds and timeout_seconds > 0 else -1
self.jdk_home = jdk_home
self.is_nailgunnable = is_nailgunnable
self.use_nailgun = use_nailgun
self.execution_slot_variable = execution_slot_variable
self.cache_scope = cache_scope

Expand Down
16 changes: 16 additions & 0 deletions src/rust/engine/Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions src/rust/engine/process_execution/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ publish = false

[dependencies]
async-trait = "0.1"
async-lock = "2.4"
walkdir = "2"
async_semaphore = { path = "../async_semaphore" }
bazel_protos = { path = "../bazel_protos" }
Expand Down
11 changes: 8 additions & 3 deletions src/rust/engine/process_execution/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@
#![allow(clippy::new_without_default, clippy::new_ret_no_self)]
// Arc<Mutex> can be more clear than needing to grok Orderings:
#![allow(clippy::mutex_atomic)]
#![type_length_limit = "43757804"]
#[macro_use]
extern crate derivative;

Expand Down Expand Up @@ -249,7 +248,13 @@ pub struct Process {

pub platform_constraint: Option<Platform>,

pub is_nailgunnable: bool,
///
/// If non-empty, the Digest of a nailgun server to use to attempt to spawn the Process.
///
/// TODO: Currently this Digest must be a subset of the `input_digest`, but we should consider
/// making it disjoint, and then automatically merging it.
///
pub use_nailgun: Digest,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there some trumping good ergonomic that outweighs the awkward ergonomic of having to pass EMPTY_DIGEST instead of None? An option seems more clear than the EMPTY_DIGEST ~magic value.

Copy link
Sponsor Member Author

@stuhood stuhood Sep 24, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the Python side this is a default argument, so users mostly don't need to worry about it. But we have in general used the "null object" pattern (with EMPTY_DIGEST as null) for digests, rather than making them optional on Process: see input_files, for example.


pub cache_scope: ProcessCacheScope,
}
Expand Down Expand Up @@ -279,7 +284,7 @@ impl Process {
append_only_caches: BTreeMap::new(),
jdk_home: None,
platform_constraint: None,
is_nailgunnable: false,
use_nailgun: hashing::EMPTY_DIGEST,
execution_slot_variable: None,
cache_scope: ProcessCacheScope::Successful,
}
Expand Down
16 changes: 15 additions & 1 deletion src/rust/engine/process_execution/src/local.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -273,6 +273,7 @@ impl super::CommandRunner for CommandRunner {
self.executor.clone(),
self.cleanup_local_dirs,
&self.work_dir_base,
(),
self.platform(),
)
.map_err(|msg| {
Expand All @@ -295,13 +296,16 @@ impl super::CommandRunner for CommandRunner {

#[async_trait]
impl CapturedWorkdir for CommandRunner {
type WorkdirToken = ();

fn named_caches(&self) -> &NamedCaches {
&self.named_caches
}

async fn run_in_workdir<'a, 'b, 'c>(
&'a self,
workdir_path: &'b Path,
_workdir_token: (),
req: Process,
_context: Context,
exclusive_spawn: bool,
Expand Down Expand Up @@ -410,6 +414,8 @@ impl CapturedWorkdir for CommandRunner {

#[async_trait]
pub trait CapturedWorkdir {
type WorkdirToken: Send;

async fn run_and_capture_workdir(
&self,
req: Process,
Expand All @@ -418,6 +424,7 @@ pub trait CapturedWorkdir {
executor: task_executor::Executor,
cleanup_local_dirs: bool,
workdir_base: &Path,
workdir_token: Self::WorkdirToken,
platform: Platform,
) -> Result<FallibleProcessResultWithPlatform, String> {
let start_time = Instant::now();
Expand Down Expand Up @@ -558,7 +565,13 @@ pub trait CapturedWorkdir {
let child_results_result = {
let child_results_future = ChildResults::collect_from(
self
.run_in_workdir(&workdir_path, req.clone(), context, exclusive_spawn)
.run_in_workdir(
&workdir_path,
workdir_token,
req.clone(),
context,
exclusive_spawn,
)
.await?,
);
if let Some(req_timeout) = req.timeout {
Expand Down Expand Up @@ -676,6 +689,7 @@ pub trait CapturedWorkdir {
async fn run_in_workdir<'a, 'b, 'c>(
&'a self,
workdir_path: &'b Path,
workdir_token: Self::WorkdirToken,
req: Process,
context: Context,
exclusive_spawn: bool,
Expand Down
Loading