After a lot of investigation, it turned out this is a memory issue with Yarn. We expected by default Yarn has 8192 MB memory; however, it is set to 2048 in configuration file and that is why Mapreduce jobs will never end (it goes to idle mode). The worst part is that the error is not very helpful either. To fix this issue you need to allocate more memory to Yarn on the node by changing the following property in yarn-site.xml
yarn.nodemanager.resource.memory-mb
5120
You need to make sure you have enough memory assigned to your sandbox. My sandbox has 8 gig of RAM and I assigned 5120 MB to Yarn.
... View more