Archive

Hunk search failed with java.io.IOException: Unknown Job error

Path Finder

I can't seem to figure out why my searches keep failing with these errors:
Note: I already have vix.mapreduce.jobhistory.address configured for the provider, but yet it's still failing

02-22-2016 12:39:18.547 DEBUG ERP.argus - SaslRpcClient - Received SASL message state: SUCCESS
02-22-2016 12:39:18.553 DEBUG ERP.argus - ClientServiceDelegate - Failed to contact AM/History for job job_1455920545730_8149 retrying..
02-22-2016 12:39:18.553 DEBUG ERP.argus - java.io.IOException: Unknown Job job_1455920545730_8149

02-22-2016 12:39:19.215 ERROR ERP.argus - SplunkMR - Error while checking whether the job is successful or not, job.name=SPLK_tslp000261_1456144316.3751_0. This error might be caused by not having configured vix.mapreduce.jobhistory.address, please configure this for the vix provider and retry.
02-22-2016 12:39:19.215 ERROR ERP.argus - java.io.IOException: Error while checking whether the job is successful or not, job.name=SPLK_tslp000261_1456144316.3751_0. This error might be caused by not having configured vix.mapreduce.jobhistory.address, please configure this for the vix provider and retry.
Tags (1)
0 Karma

Splunk Employee
Splunk Employee

Seems like the splunk queries were not storing the results of jobs in the correct location. So, when it then queried the history server for details on the complete jobs it couldn't answer and the 255 was generated. The history server simply had no data for the jobs in question. The error was not that the provider didn't have vix.mapreduce.jobhistory.address specified, it was that the history server didn't know anything about the jobs it was being asked about.

We needed to add the following lines to the provider index, and that appeared to have solved the issue

vix.mapreduce.jobhistory.done-dir = /mr-history/done
vix.mapreduce.jobhistory.intermediate-done-dir = /mr-history/tmp

Suddenly the history server could answer the query properly and the problem was resolved.

0 Karma