Archive

Issue with Hortonworks Yarn Sandbox (HDP 2) and Hunk

nhaddadkaveh_sp
Splunk Employee
Splunk Employee

It seems that HDP 2 Sandbox from Hortonworks does not work as it is with Hunk running on it. The data preview stuff works perfectly fine but Hunk gives you error when you want to run Mapreduce jobs with the following error:

Application application_1386035090053_0009 failed 2 times due to AM Container for appattempt_1386035090053_0009_000002 exited with exitCode: 1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
.Failing this attempt.. Failing the application.

Tags (1)

nhaddadkaveh_sp
Splunk Employee
Splunk Employee

After a lot of investigation, it turned out this is a memory issue with Yarn. We expected by default Yarn has 8192 MB memory; however, it is set to 2048 in configuration file and that is why Mapreduce jobs will never end (it goes to idle mode). The worst part is that the error is not very helpful either. To fix this issue you need to allocate more memory to Yarn on the node by changing the following property in yarn-site.xml

yarn.nodemanager.resource.memory-mb
5120

You need to make sure you have enough memory assigned to your sandbox. My sandbox has 8 gig of RAM and I assigned 5120 MB to Yarn.