Archive

Hunk Jobs not getting finalized Yarn HDP 2.x

Explorer

Hi,

I have a cluster with HDP 2.x setup.The data connected to the virtual index has 384007 events. When i run a normal search command all the events are retrieved and the job is getting finalized. But when i run a reporting search, only 383998 events are processed and the job is stuck at this point and not getting finalized. Can somebody help regarding the issue. Thanks.

Tags (2)

Splunk Employee
Splunk Employee

Is the resource manager running on the localhost? The following log messages indicate that we can't connect to the RM - either it is not running, not responding or we've got the wrong host

05-16-2014 09:39:59.630 INFO  ERP.TMob_Core_Metrics -  Client$Connection - Retrying connect to server: localhost:8050. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
05-16-2014 09:40:00.631 INFO  ERP.TMob_Core_Metrics -  Client$Connection - Retrying connect to server: localhost:8050. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

yet I see that HDFS is running elsewhere

05-16-2014 09:39:58.484 INFO  ERP.TMob_Core_Metrics -  SplunkMR$SplunkBaseMapper - using class=com.splunk.mr.input.SplunkLineRecordReader to process split=hdfs://RHEL6:9000/user/hadoop2/trace_log:0+134217728
0 Karma

Splunk Employee
Splunk Employee

Ok, does that mean that the RM settings for the provider are incorrect then? What does your indexes.conf look like?

0 Karma

Explorer

Yes the RM is running on 8040.

0 Karma

Splunk Employee
Splunk Employee

Is RM running on port 8050? The search without the head command causes Hunk to spawn a MapReduce/Yarn job thus the need for communicating with RM.

0 Karma

Explorer

Hi,

Yes the resource manager is running on localhost. I get this error only while running the command index= | stats count. But when i run the query like index=|head 384007|stats count , the job executes successfully without any issue.

0 Karma

Explorer

The link to the search.log is https://www.dropbox.com/s/pz5rb7e38knqz8m/search.log

Thanks.

Splunk Employee
Splunk Employee

Can you please send us a link to the contents of search.log? You can get to it by clicking on "Job" dropdown (above timeline to the right) >> "Inspect Job" >> "search.log" (bottom of page)