Splunk Search

Why is Hunk returning incomplete results for searches running over a few hours?

Splunk Employee
Splunk Employee

Splunk = Hunk 6.2.8 and Hunk 6.3.3
Hadoop = HDP 2.3.x

Symptoms = Searches don't return some results. On an example data set (JSON files compressed with snappy) it was observed that tiny searches (in the range of minutes/hours) would return ok. However, anything over a few hours would return incomplete results (count returned for affected time periods would be zero).

Errors = Yarn logs show:
"Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143"

0 Karma
1 Solution

Splunk Employee
Splunk Employee

Notes = The Yarn log is a generic error. Containers can be killed for many reasons. For example, if you specify an incorrect script path for a hadoop job, the timeout in the operation results in the same error.
Bugs = ERP-1847, ERP-1858
Fix/Workaround = vix.mapred.output.compress = false

View solution in original post

0 Karma

Splunk Employee
Splunk Employee

Notes = The Yarn log is a generic error. Containers can be killed for many reasons. For example, if you specify an incorrect script path for a hadoop job, the timeout in the operation results in the same error.
Bugs = ERP-1847, ERP-1858
Fix/Workaround = vix.mapred.output.compress = false

View solution in original post

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!