A workload management trigger executes an action when the condition defined in the
trigger expression is met. You can use all counters exposed by Apache Tez, the file system, and
Hive with workload management. The following types of counters are available: Directed acyclic
graph (DAG), file system, input split, Hive, job, shuffle error, and task counters.
Table 1. DAG counters
| Counter Name |
Aggregation Level |
| AM_CPU_MILLISECONDS |
AM |
| AM_GC_TIME_MILLIS |
AM |
| DATA_LOCAL_TASKS |
DAG |
| FALLOW_SLOTS_MILLIS_TASKS |
DAG |
| NUM_FAILED_TASKS |
DAG |
| NUM_FAILED_UBERTASKS |
DAG |
| NUM_KILLED_TASKS |
DAG |
| NUM_SUCCEEDED_TASKS |
DAG |
| NUM_UBER_SUBTASKS |
DAG |
| OTHER_LOCAL_TASKS |
DAG |
| RACK_LOCAL_TASKS |
DAG |
| SLOTS_MILLIS_TASKS |
DAG |
| TOTAL_LAUNCHED_TASKS |
DAG |
| TOTAL_LAUNCHED_UBERTASKS |
DAG |
Table 2. File system counters
| Counter Name |
Aggregation Level |
Comment |
| BYTES_READ |
DAG |
Supports all file system schemes. If S3A then counter must be prefixed with
"S3A_". Example: S3A_FILE_BYTES_READ |
| BYTES_WRITTEN |
DAG |
|
| FILE_BYTES_READ |
DAG |
Local file system |
| FILE_BYTES_WRITTEN |
DAG |
Local file system |
| HDFS_BYTES_READ |
DAG |
|
| HDFS_BYTES_WRITTEN |
DAG |
|
| LARGE_READ_OPS |
DAG |
|
| READ_OPS |
DAG |
|
| WRITE_OPS |
DAG |
|
Table 4. Hive counters
| Counter Name |
Aggregation Level |
Comment |
| CREATED_DYNAMIC_PARTITIONS |
DAG |
|
| CREATED_FILES |
DAG |
|
| DAG_TOTAL_TASKS |
DAG |
Sum of all tasks launched for the DAG |
| DESERIALIZE_ERRORS |
DAG |
|
| ELAPSED_TIME_MS |
QUERY |
Time from start of query submission to end of query execution |
| EXECUTION_TIME_MS |
DAG |
Time from start of DAG to end of DAG |
| RECORDS_IN |
DAG |
|
| RECORDS_OUT |
DAG |
|
| RECORDS_OUT_INTERMEDIATE |
DAG |
|
| VERTEX_TOTAL_TASKS |
VERTEX |
Equivalent to task paraellelism of particular vertex |
Table 5. Job counters
| Counter Name |
Aggregation Level |
| DATA_LOCAL_MAPS |
DAG |
| FALLOW_SLOTS_MILLIS_MAPS |
DAG |
| FALLOW_SLOTS_MILLIS_REDUCES |
DAG |
| NUM_FAILED_MAPS |
DAG |
| NUM_FAILED_REDUCES |
DAG |
| NUM_FAILED_UBERTASKS |
DAG |
| NUM_KILLED_MAPS |
DAG |
| NUM_KILLED_REDUCES |
DAG |
| NUM_UBER_SUBMAPS |
DAG |
| NUM_UBER_SUBREDUCES |
DAG |
| OTHER_LOCAL_MAPS |
DAG |
| RACK_LOCAL_MAPS |
DAG |
| SLOTS_MILLIS_MAPS |
DAG |
| SLOTS_MILLIS_REDUCES |
DAG |
| TOTAL_LAUNCHED_MAPS |
DAG |
| TOTAL_LAUNCHED_REDUCES |
DAG |
| TOTAL_LAUNCHED_UBERTASKS |
DAG |
Table 6. Shuffle error counters
| Counter Name |
Aggregation Level |
| BAD_ID |
DAG |
| CONNECTION |
DAG |
| IO_ERROR |
DAG |
| WRONG_LENGTH |
DAG |
| WRONG_MAP |
DAG |
| WRONG_REDUCE |
DAG |
Table 7. Task counters
| Counter Name |
Aggregation Level |
| ADDITIONAL_SPILL_COUNT |
DAG |
| ADDITIONAL_SPILLS_BYTES_READ |
DAG |
| ADDITIONAL_SPILLS_BYTES_WRITTEN |
DAG |
| COMBINE_INPUT_RECORDS |
DAG |
| COMBINE_OUTPUT_RECORDS |
DAG |
| COMMITTED_HEAP_BYTES |
DAG |
| CPU_MILLISECONDS |
DAG |
| FIRST_EVENT_RECEIVED |
DAG |
| GC_TIME_MILLIS |
DAG |
| INPUT_GROUPS |
DAG |
| INPUT_RECORDS_PROCESSED |
DAG |
| INPUT_SPLIT_LENGTH_BYTES |
DAG |
| LAST_EVENT_RECEIVED |
DAG |
| MERGE_PHASE_TIME |
DAG |
| MERGED_MAP_OUTPUTS |
DAG |
| NUM_DISK_TO_DISK_MERGES |
DAG |
| NUM_FAILED_SHUFFLE_INPUTS |
DAG |
| NUM_MEM_TO_DISK_MERGES |
DAG |
| NUM_SHUFFLED_INPUTS |
DAG |
| NUM_SKIPPED_INPUTS |
DAG |
| NUM_SPECULATIONS |
DAG |
| OUTPUT_BYTES |
DAG |
| OUTPUT_BYTES_PHYSICAL |
DAG |
| OUTPUT_BYTES_WITH_OVERHEAD |
DAG |
| OUTPUT_LARGE_RECORDS |
DAG |
| OUTPUT_RECORDS |
DAG |
| PHYSICAL_MEMORY_BYTES |
DAG |
| REDUCE_INPUT_GROUPS |
DAG |
| REDUCE_INPUT_RECORDS |
DAG |
| REDUCE_OUTPUT_RECORDS |
DAG |
| REDUCE_SKIPPED_GROUPS |
DAG |
| REDUCE_SKIPPED_RECORDS |
DAG |
| SHUFFLE_BYTES |
DAG |
| SHUFFLE_BYTES_DECOMPRESSED |
DAG |
| SHUFFLE_BYTES_DISK_DIRECT |
DAG |
| SHUFFLE_BYTES_TO_DISK |
DAG |
| SHUFFLE_BYTES_TO_MEM |
DAG |
| SHUFFLE_CHUNK_COUNT |
DAG |
| SHUFFLE_PHASE_TIME |
DAG |
| SKIPPED_RECORDS |
DAG |
| SPILLED_RECORDS |
DAG |
| SPLIT_RAW_BYTES |
DAG |
| VIRTUAL_MEMORY_BYTES |
DAG |
