Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Redact passwords from tasks fetched from the TaskQueue #16182

Merged
merged 4 commits into from
Mar 23, 2024
Merged

Redact passwords from tasks fetched from the TaskQueue #16182

merged 4 commits into from
Mar 23, 2024

Conversation

AmatyaAvadhanula
Copy link
Contributor

Fixes #15882

Description

Active tasks fetched from TaskQueue's in memory state may reveal sensitive information.

This doesn't happen for payloads fetched from the metadata store, as they are read using a json mapper with an added mixin for redacting credentials.

This PR leans on a similar change in the TaskQueue where a payload is serialized and then deserialized.

Release note

Key changed/added classes in this PR
  • TaskQueue

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

kfaraz
kfaraz reviewed Mar 22, 2024
View reviewed changes
Copy link
Contributor

@kfaraz kfaraz left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix.
I wonder if it would be better to just do the redaction while putting an item in the tasks map. That way the serde would have to be done only once.
Edit: On second thought, doing the redaction while putting in the map is not a viable option as this same Task instance is sent across the wire to MMs/indexers.

indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskQueue.java Outdated Show resolved Hide resolved
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskQueue.java Outdated Show resolved Hide resolved
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskQueue.java Outdated Show resolved Hide resolved
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskQueue.java Outdated Show resolved Hide resolved
kfaraz
kfaraz reviewed Mar 22, 2024
View reviewed changes
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskQueue.java Outdated
@@ -122,6 +121,7 @@ public class TaskQueue
private final TaskActionClientFactory taskActionClientFactory;
private final TaskLockbox taskLockbox;
private final ServiceEmitter emitter;
private final ObjectMapper PASSWORD_REDACTING_MAPPER;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rename since it is not a constant anymore.

Suggested change
private final ObjectMapper PASSWORD_REDACTING_MAPPER;
private final ObjectMapper passwordRedactingMapper;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskQueue.java Outdated
Comment on lines 1004 to 1006
throw DruidException.forPersona(DruidException.Persona.USER)
.ofCategory(DruidException.Category.RUNTIME_FAILURE)
.build(e, "Failed to serialize or deserialize task.");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this exception should be targeted either at developer or operator, not the user persona.

Suggested change
throw DruidException.forPersona(DruidException.Persona.USER)
.ofCategory(DruidException.Category.RUNTIME_FAILURE)
.build(e, "Failed to serialize or deserialize task.");
throw DruidException.forPersona(DruidException.Persona.OPERATOR)
.ofCategory(DruidException.Category.RUNTIME_FAILURE)
.build(e, "Failed to serialize or deserialize task[%s].", task.getId());

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskStorageQueryAdapter.java Outdated
@@ -114,7 +114,7 @@ public Optional<Task> getTask(final String taskid)
return activeTask;
}
}
catch (JsonProcessingException e) {
catch (DruidException e) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we swallowing this exception?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought it might be helpful to fallback to the metadata store (TaskStorage) in such a case

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have to do one of two things:
Either (1) Throw JsonProcessingException from TaskQueue.getActiveTask(), catch it here, log it and fallback to using TaskStorage (catch block should have a comment to this effect). In this case, TaskQueue.getActiveTask need not log the exception.
Or (2) Throw DruidException from TaskQueue.getActiveTask() and don't catch it anywhere.

I would prefer the latter as exception during serde shouldn't happen, and it is best to notify the operator about it as soon as possible.
Also if there is an issue deserializing the task, the task storage would be likely to encounter the same issue as it is the same mapper and the same Task object.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

indexing-service/src/test/java/org/apache/druid/indexing/overlord/TaskQueueTest.java Outdated
taskQueue.start();
taskQueue.add(taskWithPassword);

final Optional<Task> taskFromTaskStorage = taskStorage.getTask(taskWithPassword.getId());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
final Optional<Task> taskFromTaskStorage = taskStorage.getTask(taskWithPassword.getId());
final Optional<Task> taskInStorage = taskStorage.getTask(taskWithPassword.getId());

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

indexing-service/src/test/java/org/apache/druid/indexing/overlord/TaskQueueTest.java Outdated
Assert.assertFalse(mapper.writeValueAsString(taskFromTaskStorage.get()).contains(password));


final Optional<Task> taskFromTaskQueue = taskQueue.getActiveTask(taskWithPassword.getId());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
final Optional<Task> taskFromTaskQueue = taskQueue.getActiveTask(taskWithPassword.getId());
final Optional<Task> taskInQueue = taskQueue.getActiveTask(taskWithPassword.getId());

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

indexing-service/src/test/java/org/apache/druid/indexing/overlord/TaskQueueTest.java Outdated

final Optional<Task> taskFromTaskQueue = taskQueue.getActiveTask(taskWithPassword.getId());
Assert.assertTrue(taskFromTaskQueue.isPresent());
Assert.assertNotNull(taskFromTaskQueue.get());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assign the result of optional.get() to a new variable and then do the assertions.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

indexing-service/src/test/java/org/apache/druid/indexing/overlord/TaskQueueTest.java Show resolved Hide resolved
kfaraz
kfaraz reviewed Mar 22, 2024
View reviewed changes
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskQueue.java Outdated
@@ -152,6 +157,7 @@ public class TaskQueue
private final AtomicInteger statusUpdatesInQueue = new AtomicInteger();
private final AtomicInteger handledStatusUpdates = new AtomicInteger();


Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: extra line?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed

@AmatyaAvadhanula AmatyaAvadhanula merged commit cfa2a90 into apache:master Mar 23, 2024
85 checks passed
@AmatyaAvadhanula
Copy link
Contributor Author

Thank you for the reviews @kfaraz @cryptoe

@adarshsanjeev adarshsanjeev added this to the 30.0.0 milestone May 6, 2024
@adarshsanjeev adarshsanjeev mentioned this pull request May 28, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cryptoe cryptoe cryptoe left review comments

@kfaraz kfaraz kfaraz approved these changes

Assignees
No one assigned
Labels
Area - Ingestion
Projects
None yet
Milestone
30.0.0
Development

Successfully merging this pull request may close these issues.

DefaultPasswordProvider shows password when task is running
4 participants
@AmatyaAvadhanula @cryptoe @kfaraz @adarshsanjeev