Splunk Search

Hunk Jobs not getting finalized Yarn HDP 2.x

eseepnoname
Explorer

Hi,

I have a cluster with HDP 2.x setup.The data connected to the virtual index has 384007 events. When i run a normal search command all the events are retrieved and the job is getting finalized. But when i run a reporting search, only 383998 events are processed and the job is stuck at this point and not getting finalized. Can somebody help regarding the issue. Thanks.

Tags (2)

Ledion_Bitincka
Splunk Employee
Splunk Employee

Is the resource manager running on the localhost? The following log messages indicate that we can't connect to the RM - either it is not running, not responding or we've got the wrong host

05-16-2014 09:39:59.630 INFO  ERP.TMob_Core_Metrics -  Client$Connection - Retrying connect to server: localhost:8050. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
05-16-2014 09:40:00.631 INFO  ERP.TMob_Core_Metrics -  Client$Connection - Retrying connect to server: localhost:8050. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

yet I see that HDFS is running elsewhere

05-16-2014 09:39:58.484 INFO  ERP.TMob_Core_Metrics -  SplunkMR$SplunkBaseMapper - using class=com.splunk.mr.input.SplunkLineRecordReader to process split=hdfs://RHEL6:9000/user/hadoop2/trace_log:0+134217728
0 Karma

Ledion_Bitincka
Splunk Employee
Splunk Employee

Ok, does that mean that the RM settings for the provider are incorrect then? What does your indexes.conf look like?

0 Karma

eseepnoname
Explorer

Yes the RM is running on 8040.

0 Karma

Ledion_Bitincka
Splunk Employee
Splunk Employee

Is RM running on port 8050? The search without the head command causes Hunk to spawn a MapReduce/Yarn job thus the need for communicating with RM.

0 Karma

eseepnoname
Explorer

Hi,

Yes the resource manager is running on localhost. I get this error only while running the command index= | stats count. But when i run the query like index=|head 384007|stats count , the job executes successfully without any issue.

0 Karma

eseepnoname
Explorer

The link to the search.log is https://www.dropbox.com/s/pz5rb7e38knqz8m/search.log

Thanks.

Ledion_Bitincka
Splunk Employee
Splunk Employee

Can you please send us a link to the contents of search.log? You can get to it by clicking on "Job" dropdown (above timeline to the right) >> "Inspect Job" >> "search.log" (bottom of page)

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...