Limiting the usage Counters in Hadoop
Besides the JobCounter and the TaskCounter counters which Hadoop framework maintains, it's also possible to define custom counters for application level statistics.
Counters can be incremented using the Reporter for the Old MapReduce API or by using theContext using the New MapReduce API. These counters are sent to the TaskTracker and the TaskTracker will send to the JobTracker and the JobTracker will consolidate the Counters to produce a holistic view for the complete Job.
There is a chance that a rogue Job creates millions of counters and since these counters are stored in the JobTracker, there is a better chance that JobTracker will go OOM. To avoid such a scenario the number of counters that can be created per Job are limited by the Hadoop framework.
The following code is in the Counters.java. Note that this code is in the 20.203, 20.204 and 20.205 (now called 1.0) releases. Also note that some of the parameters are configurable and some are not.
In trunk and 0.23 release the below code is there in the MRJobConfig.java. Note that the parameters are configurable.
The above mentioned configuration parameters are not mentioned in the release documentation an so thought it would be worth mentioning as a blog entry.
I would like to call it a hidden jewel :)
Counters can be incremented using the Reporter for the Old MapReduce API or by using theContext using the New MapReduce API. These counters are sent to the TaskTracker and the TaskTracker will send to the JobTracker and the JobTracker will consolidate the Counters to produce a holistic view for the complete Job.
There is a chance that a rogue Job creates millions of counters and since these counters are stored in the JobTracker, there is a better chance that JobTracker will go OOM. To avoid such a scenario the number of counters that can be created per Job are limited by the Hadoop framework.
The following code is in the Counters.java. Note that this code is in the 20.203, 20.204 and 20.205 (now called 1.0) releases. Also note that some of the parameters are configurable and some are not.
1
2
3
4
5
6
7
8
9
10
11
12
| /** limit on the size of the name of the group **/private static final int GROUP_NAME_LIMIT = 128;/** limit on the size of the counter name **/private static final int COUNTER_NAME_LIMIT = 64;private static final JobConf conf = new JobConf();/** limit on counters **/public static int MAX_COUNTER_LIMIT = conf.getInt("mapreduce.job.counters.limit", 120);/** the max groups allowed **/static final int MAX_GROUP_LIMIT = 50; |
In trunk and 0.23 release the below code is there in the MRJobConfig.java. Note that the parameters are configurable.
1
2
3
4
5
6
7
8
9
10
11
| public static final String COUNTERS_MAX_KEY = "mapreduce.job.counters.max";public static final int COUNTERS_MAX_DEFAULT = 120;public static final String COUNTER_GROUP_NAME_MAX_KEY = "mapreduce.job.counters.group.name.max";public static final int COUNTER_GROUP_NAME_MAX_DEFAULT = 128;public static final String COUNTER_NAME_MAX_KEY = "mapreduce.job.counters.counter.name.max";public static final int COUNTER_NAME_MAX_DEFAULT = 64;public static final String COUNTER_GROUPS_MAX_KEY = "mapreduce.job.counters.groups.max";public static final int COUNTER_GROUPS_MAX_DEFAULT = 50; |
The above mentioned configuration parameters are not mentioned in the release documentation an so thought it would be worth mentioning as a blog entry.
I would like to call it a hidden jewel :)
Comments
Post a Comment
thank you for your feedback