Monitoring Splunk

Why are _internal logs from heavy forwarder(HF) not getting to indexers after a Splunkd restart but _audit are?

Communicator

All of a sudden, _internal logs from HF stopped coming to indexers after a Splunkd restart. But, i see _audit logs making it to indexers. And, I see splunkd.log on HF is growing. There is no change in inputs.conf or outputs.conf before restart. What could be the reason?

0 Karma

Influencer

Use btool to check on your inputs for splunkd.log files:

/opt/splunk/bin/splunk btool inputs list --debug | grep -B 5 log/splunk

If there is no TCP_ROUTING sending those to somewhere strange, check the /opt/splunk/var/log on the HF to check the modtime of splunkd.

More, do a tail -f on splunkd.log to check if these are being written

Finally, on your Search Head do a | tstats count where host=yourhf by index, _time

and check if something else has stopped meanwhile from that host

0 Karma

Influencer

@Rob2520 please accept an answer if it solved/helped it and upvote it. Otherwise let us know how can we help further

0 Karma

SplunkTrust
SplunkTrust

Check props.conf and/or transforms.conf if there is any filtering or routing configured. I know that _audit is not effected by those settings and therefore reaches your indexer. Also these kind of configuration changes need a Splunk restart to take effect.

cheers, MuS

0 Karma

Communicator

MuS, i don't see props or transforms related to splunkd logs.

0 Karma