How do we analyze indexers/search head log?

[makincerdas]

I have a working environment using index discovery.

While doing command 'tail -f' on idx01 (Indexer - master box), I notice the following logs.

idx01 (tail -f /opt/splunk/var/log/splunk/splunkd.log)

10-19-2016 20:59:27.187 +0000 WARN DistributedPeerManager - Unable to distribute to peer named idx03 at uri https://10.200.2.35:8089 because replication was unsuccessful. replicationStatus Failed failure info: failed_because_HTTP_REPLY_READ_FAILURE
10-19-2016 20:59:27.187 +0000 WARN DistributedPeerManager - Unable to distribute to peer named idx04 at uri https://10.200.2.38:8089 because replication was unsuccessful. replicationStatus Failed failure info: failed_because_HTTP_CONNECTION_FAILURE
10-19-2016 20:59:38.186 +0000 INFO TcpOutputProc - Connected to idx=10.200.2.36:9997
10-19-2016 20:59:46.316 +0000 WARN DistributedBundleReplicationManager - Asynchronous bundle replication to 3 peer(s) succeeded; however it took too long (longer than 10 seconds): elapsed_ms=19129, tar_elapsed_ms=3726, bundle_file_size=75900KB, replication_id=1476910767, replication_reason="async replication allowed"

I did search using this string 'Failed failure info' on splunk, but found nothing.

I am not sure if logs on indexers are parsed and indexed.
What is the best practice to monitor/analyze logs on indexers/search head boxes?

Thank you.

[makincerdas]

I did not use 'index=_internal' on earlier attempt.
It works by entering the following search, using 'index=_internal'

index="_internal" Failed failure info source="/opt/splunk/var/log/splunk/splunkd.log"

Thank you @TStrauch

[TStrauch · Accepted solution]

Hi,

the splunkd.log on your indexers should be indexed automatically. You can find them by searching index=_internal. There you will find all internal splunk logdata.

For getting the internal logs of your searchhead to the indexer tier look at this. This works similar for your master node.

http://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Forwardsearchheaddata

kind regards