topic Re: Hunk Jobs not getting finalized Yarn HDP 2.x in Splunk Search

Hunk Jobs not getting finalized Yarn HDP 2.x

eseepnoname — Mon, 19 May 2014 17:44:13 GMT

Hi,

I have a cluster with HDP 2.x setup.The data connected to the virtual index has 384007 events. When i run a normal search command all the events are retrieved and the job is getting finalized. But when i run a reporting search, only 383998 events are processed and the job is stuck at this point and not getting finalized. Can somebody help regarding the issue. Thanks.

Re: Hunk Jobs not getting finalized Yarn HDP 2.x

Ledion_Bitincka — Mon, 19 May 2014 17:53:30 GMT

Can you please send us a link to the contents of search.log? You can get to it by clicking on "Job" dropdown (above timeline to the right) >> "Inspect Job" >> "search.log" (bottom of page)

Re: Hunk Jobs not getting finalized Yarn HDP 2.x

eseepnoname — Mon, 19 May 2014 18:03:30 GMT

The link to the search.log is https://www.dropbox.com/s/pz5rb7e38knqz8m/search.log

Thanks.

Re: Hunk Jobs not getting finalized Yarn HDP 2.x

Ledion_Bitincka — Mon, 19 May 2014 18:27:21 GMT

Is the resource manager running on the localhost? The following log messages indicate that we can't connect to the RM - either it is not running, not responding or we've got the wrong host

05-16-2014 09:39:59.630 INFO  ERP.TMob_Core_Metrics -  Client$Connection - Retrying connect to server: localhost:8050. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
05-16-2014 09:40:00.631 INFO  ERP.TMob_Core_Metrics -  Client$Connection - Retrying connect to server: localhost:8050. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

yet I see that HDFS is running elsewhere

05-16-2014 09:39:58.484 INFO  ERP.TMob_Core_Metrics -  SplunkMR$SplunkBaseMapper - using class=com.splunk.mr.input.SplunkLineRecordReader to process split=hdfs://RHEL6:9000/user/hadoop2/trace_log:0+134217728

Re: Hunk Jobs not getting finalized Yarn HDP 2.x

eseepnoname — Mon, 19 May 2014 18:50:31 GMT

Hi,

Yes the resource manager is running on localhost. I get this error only while running the command index= | stats count. But when i run the query like index=|head 384007|stats count , the job executes successfully without any issue.

Re: Hunk Jobs not getting finalized Yarn HDP 2.x

Ledion_Bitincka — Mon, 19 May 2014 19:23:56 GMT

Is RM running on port 8050? The search without the head command causes Hunk to spawn a MapReduce/Yarn job thus the need for communicating with RM.

Re: Hunk Jobs not getting finalized Yarn HDP 2.x

eseepnoname — Mon, 19 May 2014 20:18:02 GMT

Yes the RM is running on 8040.

Re: Hunk Jobs not getting finalized Yarn HDP 2.x

Ledion_Bitincka — Tue, 20 May 2014 00:12:33 GMT

Ok, does that mean that the RM settings for the provider are incorrect then? What does your indexes.conf look like?