Archive
Highlighted

How do we analyze indexers/search head log?

New Member

I have a working environment using index discovery.

While doing command 'tail -f' on idx01 (Indexer - master box), I notice the following logs.

idx01 (tail -f /opt/splunk/var/log/splunk/splunkd.log)

10-19-2016 20:59:27.187 +0000 WARN DistributedPeerManager - Unable to distribute to peer named idx03 at uri https://10.200.2.35:8089 because replication was unsuccessful. replicationStatus Failed failure info: failedbecauseHTTPREPLYREADFAILURE
10-19-2016 20:59:27.187 +0000 WARN DistributedPeerManager - Unable to distribute to peer named idx04 at uri https://10.200.2.38:8089 because replication was unsuccessful. replicationStatus Failed failure info: failed
becauseHTTPCONNECTIONFAILURE
10-19-2016 20:59:38.186 +0000 INFO TcpOutputProc - Connected to idx=10.200.2.36:9997
10-19-2016 20:59:46.316 +0000 WARN DistributedBundleReplicationManager - Asynchronous bundle replication to 3 peer(s) succeeded; however it took too long (longer than 10 seconds): elapsed
ms=19129, tarelapsedms=3726, bundlefilesize=75900KB, replicationid=1476910767, replicationreason="async replication allowed"

I did search using this string 'Failed failure info' on splunk, but found nothing.

I am not sure if logs on indexers are parsed and indexed.
What is the best practice to monitor/analyze logs on indexers/search head boxes?

Thank you.

Tags (1)
0 Karma
Highlighted

Re: How do we analyze indexers/search head log?

Communicator

Hi,

the splunkd.log on your indexers should be indexed automatically. You can find them by searching index=_internal. There you will find all internal splunk logdata.

For getting the internal logs of your searchhead to the indexer tier look at this. This works similar for your master node.

http://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Forwardsearchheaddata

kind regards

View solution in original post

Highlighted

Re: How do we analyze indexers/search head log?

New Member

I did not use 'index=internal' on earlier attempt.
It works by entering the following search, using 'index=
internal'

index="_internal" Failed failure info source="/opt/splunk/var/log/splunk/splunkd.log"

Thank you @TStrauch

0 Karma