[SPARK-12149] [Web UI] Executor UI improvement suggestions - Color UI #10154

ajbozarth · 2015-12-05T02:47:38Z

Added color coding to the Executors page for Active Tasks, Failed Tasks, Completed Tasks and Task Time.

Active Tasks is shaded blue with it's range based on percentage of total cores used.
Failed Tasks is shaded red ranging over the first 10% of total tasks failed
Completed Tasks is shaded green ranging over 10% of total tasks including failed and active tasks, but only when there are active or failed tasks on that executor.
Task Time is shaded red when GC Time goes over 10% of total time with it's range directly corresponding to the percent of total time.

ajbozarth · 2015-12-05T02:50:47Z

Few discussion issues to raise:

I chose to use opacity (alpha channel) to range the colors, what do people think of this? I felt it was simpler than figuring out a range of rgb.
I chose a 10% threshold for the GC color appearing, is that a good threshold or should I change it?
What do people think of my choices for determining the color ranges?

srowen · 2015-12-05T15:17:16Z

core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala

+      // activeTasks range from 0 to all cores
+      activeTasksAlpha = (info.activeTasks.toDouble / info.totalCores) * 0.5 + 0.5
+    }
+    if (info.totalTasks > 0) {


You can also assign these like

val (failedTasksAlpha, completedTasksAlpha) = if (...) { (..., ...) } else { (1.0, 1.0) }

but whether that's clearer is a matter of taste.
It might be better to cap alpha at 1.0 just for tidiness.

I wanted to do that but the style guide wasn't clear on whether it was allow, I'll update it since it's cleaner. And I'll add the 1.0 cap, I left it off originally because the code was getting busy and hard to read originally.

SparkQA · 2015-12-05T15:37:55Z

Test build #2173 has finished for PR 10154 at commit d630cac.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2015-12-06T01:37:09Z

Can you post screenshots?

holdenk · 2015-12-06T21:38:27Z

core/src/test/resources/HistoryServerExpectations/executor_list_json_expectation.json

@@ -4,11 +4,13 @@
  "rddBlocks" : 8,
  "memoryUsed" : 28000128,
  "diskUsed" : 0,
+  "totalCores" : 0,


The expected number of cores is 0?

I updated that file according to the javadocs for HistoryServerSuite, the test that uses it. I would assume it's 0 since it's the driver and the driver has no cores assigned to run tasks.

ajbozarth · 2015-12-07T19:27:02Z

Here's a few screen shots, note that I lowered the threshold and range for the GC Time Color so you can see it easier for this example (1% threshold with color range over first 5% instead of 10% threshold with range over first 50%)

srowen · 2015-12-07T19:40:16Z

Thanks for the screenshots. The color really 'pops', so my concern is just making sure it doesn't get overwhelming.

I can see using the red to very clearly call attention to failed things, or excessive GC. It also sort of feels write to call attention to active things while they're active. (If green = good and red = bad and blue = neutral, is blue more appropriate for an active task? it could turn out to fail or complete. shrug.)

I wasn't as sure about coloring the completed tasks column therefore, but it's a matter of taste.

I kind of like how this looks, but if anyone else is concerned about the amount of color, I suppose it's coherent to make the text colored instead of the background.

ajbozarth · 2015-12-07T22:17:50Z

I ran the MiMa tests locally (I didn't know they didn't run before) and they're failing because I changed api.scala and added two vals to ExecutorSummary. How do I go about fixing the MiMa failure?

srowen · 2015-12-07T22:20:05Z

It must be a false positive or else some problem with how mima got invoked. I don't see any API changes here. If they're "real" false positives you'll need to add an exclusion but it tells you what to add. Let's see what Jenkins says.

SparkQA · 2015-12-07T22:35:14Z

Test build #2178 has finished for PR 10154 at commit 2887aca.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

… GC time

ajbozarth · 2015-12-09T00:32:25Z

added the MiMa exclude, please advise if I did it correctly

holdenk · 2015-12-10T19:46:59Z

core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala

@@ -33,11 +33,13 @@ private[ui] case class ExecutorSummaryInfo(
    rddBlocks: Int,
    memoryUsed: Long,
    diskUsed: Long,
+    totalCores: Int,


So the comment for this case class says it isn't used anymore - do we really need to update it?

Did some checks, MiMa exclude is still needed even without this change and this change did not affect any code, therefore I've removed the change.

ajbozarth · 2015-12-16T22:49:40Z

Is there a way someone could get the tests to run to check my change? I assumed they would run on my commit, but they didn't.

srowen · 2015-12-17T08:06:06Z

Jenkins test this please

srowen · 2015-12-17T08:06:19Z

Jenkins add to whitelist

SparkQA · 2015-12-17T10:19:06Z

Test build #47913 has finished for PR 10154 at commit 3208d85.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ajbozarth · 2015-12-17T18:42:46Z

Thanks @srowen , Is there anyone we should pull into this to get more opinions on the change?

srowen · 2015-12-21T11:51:47Z

My only reservations are: active vs complete being colored green vs blue instead of the other way around, and also coloring the complete column at all, since it would always be colored after starting.

ajbozarth · 2015-12-21T19:12:01Z

I'll switch the green and blue and post some screen shots to see the difference.

As for the completed column always being blue:
I could make it conditional so it's only blue if there are either active or failed tasks to compare too, but I feel like that would end up looking odd when some executors' completed tasks are colored and some aren't.
Or I could remove the coloring from completed tasks entirely, I'll post some screenshots of this as well before pushing any changes.

ajbozarth · 2015-12-21T20:26:03Z

There's three screenshots of what switching the blue and green would look like, I actually think I like it better.

ajbozarth · 2015-12-21T21:00:38Z

And here's screenshots of completed only being colored when comparing to active or failed, examples with both color choices.

Personally I don't know how I feel about this, I think it looks better visually, but I'm not sure about usability.

ajbozarth · 2016-01-19T18:08:51Z

Thanks, I only tested using a standalone cluster, I have Hadoop and Yarn set up on my remote testing cluster so I can test that now.

tgravescs · 2016-01-19T19:15:33Z

So after seeing this run I'm not sure how useful the Completed Task column shading it. Especially when there aren't any tasks running. I think it draws your attention to that column for no reason. So I think I would like to see the shading on completed tasks removed unless someone has a strong use case for it, thoughts?

I still need to try out the failed and GC time.

ajbozarth · 2016-01-19T19:36:06Z

I had an implementation a handful of commits ago that only shaded completed when either active or failed was also colored for that row, allowing for quick comparison. What do you think of that option (there are screen shots of it above)? Also if we got rid of coloring completed tasks do we want to switch active back to green or leave it as blue?

ajbozarth · 2016-01-19T22:30:13Z

Tested it with Yarn and it behaves the same way as Standalone

tgravescs · 2016-01-19T22:41:41Z

thanks, I saw the above changes for only showing completed when active or failed but again I'm not sure how useful it is. I think just dropping the color for now makes sense. I think leave the color blue in active.

If we were to keep completed tasks I think it makes more sense to shade vs other executors. The times I can think of wanting to know completed tasks is different is perhaps when that executor is being slow or skewed data. So if it doesn't have as many completed tasks as other executors. I don't think completed vs total on same executor is really that useful. But I think for now we just remove it.

ajbozarth · 2016-01-19T23:45:40Z

Screen shot of my incoming update:

SparkQA · 2016-01-20T01:23:03Z

Test build #49727 has finished for PR 10154 at commit 2f54a33.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

ajbozarth · 2016-01-20T03:59:55Z

retest please

tgravescs · 2016-01-20T15:24:00Z

Jenkins, test this please

SparkQA · 2016-01-20T17:24:42Z

Test build #49788 has finished for PR 10154 at commit 2f54a33.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tgravescs · 2016-01-22T17:16:43Z

I haven't forgot about this, just busy, I want to take one last pass over this.

tgravescs · 2016-01-25T14:27:31Z

core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala

+    // failedTasks range max at 10% failure, alpha max = 1
+    val failedTasksAlpha =
+      if (totalTasks > 0) {
+        math.min (10 * failedTasks.toDouble / totalTasks, 1) * 0.5 + 0.5


remove space after min

tgravescs · 2016-01-25T15:01:33Z

A few minor nits, otherwise looks good.

ajbozarth · 2016-01-25T16:55:55Z

Fixed the little issues, don't know when the space after min got there though. And this also showed me my auto-format in IntelliJ isn't 100% correct.

SparkQA · 2016-01-25T18:31:11Z

Test build #49996 has finished for PR 10154 at commit 9293c41.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

ajbozarth · 2016-01-25T18:33:14Z

retest this please

tgravescs · 2016-01-25T18:35:29Z

Jenkins, test this please

SparkQA · 2016-01-25T20:25:21Z

Test build #50000 has finished for PR 10154 at commit 9293c41.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tgravescs · 2016-01-25T20:43:50Z

+1 committed this, thanks @ajbozarth

rxin · 2016-10-18T21:31:24Z

So I looked at the UI today and I have to say the color code is extremely confusing. On a normal cluster I saw a bunch of reds, which is usually reserved for errors.

(I'm going to leave a comment on the JIRA ticket too)

ajbozarth · 2016-10-18T21:44:11Z

@rxin The reds only show up on failed tasks or a high gc time, what would you expect from each color? Also this was a while ago (and one of my first prs) so I'm willing to revisit how this was done. If we want to open a new JIRA to revisit it I'll work on it.

(responding here since this is where all the reviewers are looped in)

srowen · 2016-10-19T08:54:36Z

Yes, I have not noticed the color to be confusing. What type of state are you looking at @rxin? it could be that with enough activity, failures are inevitable, and marking anything > 0 failures as red is a lot.

srowen reviewed Dec 5, 2015
View reviewed changes

holdenk reviewed Dec 6, 2015
View reviewed changes

ajbozarth force-pushed the spark12149 branch from 8016acb to d630cac Compare December 7, 2015 20:38

ajbozarth added 4 commits December 8, 2015 16:21

SPARK-1832 added color to tasks on executor page

55a1c31

SPARK-12149 Cleaned up color UI code, fixed tests, added color UI for…

d9f2b82

… GC time

Addressed comments

1da5f1a

added MiMa exclude

20c7b48

ajbozarth force-pushed the spark12149 branch from 143ea61 to 20c7b48 Compare December 9, 2015 00:29

holdenk reviewed Dec 10, 2015
View reviewed changes

Comments followup

1652274

Removed Completed Tasks coloring

2f54a33

tgravescs reviewed Jan 25, 2016
View reviewed changes

Fixed some style issues

9293c41

asfgit closed this in c037d25 Jan 25, 2016

ajbozarth deleted the spark12149 branch January 25, 2016 21:02

ajbozarth mentioned this pull request Feb 3, 2016

[SPARK-3611] [Web UI] Show number of cores for each executor in application web UI #11039

Closed

panbingkun mentioned this pull request May 26, 2023

[SPARK-43576][CORE] Remove unused declarations from Core module #41218

Closed

[SPARK-12149] [Web UI] Executor UI improvement suggestions - Color UI #10154

[SPARK-12149] [Web UI] Executor UI improvement suggestions - Color UI #10154

Conversation

ajbozarth commented Dec 5, 2015

ajbozarth commented Dec 5, 2015

srowen Dec 5, 2015

Choose a reason for hiding this comment

ajbozarth Dec 7, 2015

Choose a reason for hiding this comment

SparkQA commented Dec 5, 2015

JoshRosen commented Dec 6, 2015

holdenk Dec 6, 2015

Choose a reason for hiding this comment

ajbozarth Dec 7, 2015

Choose a reason for hiding this comment

ajbozarth commented Dec 7, 2015

srowen commented Dec 7, 2015

ajbozarth commented Dec 7, 2015

srowen commented Dec 7, 2015

SparkQA commented Dec 7, 2015

ajbozarth commented Dec 9, 2015

holdenk Dec 10, 2015

Choose a reason for hiding this comment

ajbozarth Dec 11, 2015

Choose a reason for hiding this comment

ajbozarth commented Dec 16, 2015

srowen commented Dec 17, 2015

srowen commented Dec 17, 2015

SparkQA commented Dec 17, 2015

ajbozarth commented Dec 17, 2015

srowen commented Dec 21, 2015

ajbozarth commented Dec 21, 2015

ajbozarth commented Dec 21, 2015

ajbozarth commented Dec 21, 2015

ajbozarth commented Jan 19, 2016

tgravescs commented Jan 19, 2016

ajbozarth commented Jan 19, 2016

ajbozarth commented Jan 19, 2016

tgravescs commented Jan 19, 2016

ajbozarth commented Jan 19, 2016

SparkQA commented Jan 20, 2016

ajbozarth commented Jan 20, 2016

tgravescs commented Jan 20, 2016

SparkQA commented Jan 20, 2016

tgravescs commented Jan 22, 2016

tgravescs Jan 25, 2016

Choose a reason for hiding this comment

tgravescs commented Jan 25, 2016

ajbozarth commented Jan 25, 2016

SparkQA commented Jan 25, 2016

ajbozarth commented Jan 25, 2016

tgravescs commented Jan 25, 2016

SparkQA commented Jan 25, 2016

tgravescs commented Jan 25, 2016

rxin commented Oct 18, 2016

ajbozarth commented Oct 18, 2016

srowen commented Oct 19, 2016