-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add runtime stats to track per-query memory reclaim time and bytes (#…
…9196) Summary: Add two runtime stats to track operator-level memory reclaim time and reclaimed memory bytes to help analyze slow query Add operator and task memory reclaim counters and the metric rename accordingly Add documentation for the newly added runtime stats and cover the existing ones in followup Pull Request resolved: #9196 Reviewed By: mbasmanova Differential Revision: D55177631 Pulled By: xiaoxmeng fbshipit-source-id: 42534504ad9bd910a7106f805bc09c5ebf35ba36
- Loading branch information
1 parent
5578357
commit 1ed8075
Showing
9 changed files
with
104 additions
and
21 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
********** | ||
Monitoring | ||
********** | ||
|
||
.. toctree:: | ||
:maxdepth: 1 | ||
|
||
monitoring/metrics.rst | ||
monitoring/stats.rst |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,41 @@ | ||
============= | ||
Runtime Stats | ||
============= | ||
|
||
Runtime stats are used to collect the per-query velox runtime events for | ||
offline query analysis purpose. The collected stats can provide insights into | ||
the operator level query execution internals, such as how much time a query | ||
operator spent in disk spilling. The collected stats are organized in a | ||
free-form key-value for easy extension. The key is the event name and the | ||
value is defined as RuntimeCounter which is used to store and aggregate a | ||
particular event occurrences during the operator execution. RuntimeCounter has | ||
three types: kNone used to record event count, kNanos used to record event time | ||
in nanoseconds and kBytes used to record memory or storage size in bytes. It | ||
records the count of events, and the min/max/sum of the event values. The stats | ||
are stored in OperatorStats structure. The query system can aggregate the | ||
operator level stats collected from each driver by pipeline and task for | ||
analysis. | ||
|
||
Memory Arbitration | ||
------------------ | ||
.. list-table:: | ||
:widths: 40 10 50 | ||
:header-rows: 1 | ||
|
||
* - Stats Name | ||
- Type | ||
- Description | ||
* - memoryReclaimCount | ||
- kNone | ||
- The number of times that the memory arbitration to reclaim memory from | ||
an spillable operator. | ||
This stats only applies for spillable operators. | ||
* - memoryReclaimWallNanos | ||
- kNano | ||
- The memory reclaim execution time of an operator during the memory | ||
arbitration. It collects time spent on disk spilling or file write. | ||
This stats only applies for spillable operators. | ||
* - reclaimedMemoryBytes | ||
- kByte | ||
- The reclaimed memory bytes of an operator during the memory arbitration. | ||
This stats only applies for spillable operators. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters