Why is Hunk returning incomplete results for searches running over a few hours?

[bohanlon_splunk]

Splunk = Hunk 6.2.8 and Hunk 6.3.3
Hadoop = HDP 2.3.x

Symptoms = Searches don't return some results. On an example data set (JSON files compressed with snappy) it was observed that tiny searches (in the range of minutes/hours) would return ok. However, anything over a few hours would return incomplete results (count returned for affected time periods would be zero).

Errors = Yarn logs show:
"Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143"

[bohanlon_splunk · Accepted solution]

Notes = The Yarn log is a generic error. Containers can be killed for many reasons. For example, if you specify an incorrect script path for a hadoop job, the timeout in the operation results in the same error.
Bugs = ERP-1847, ERP-1858
Fix/Workaround = vix.mapred.output.compress = false