Why are _internal logs from heavy forwarder(HF) not getting to indexers after a Splunkd restart but _audit are?

[Rob2520]

All of a sudden, _internal logs from HF stopped coming to indexers after a Splunkd restart. But, i see _audit logs making it to indexers. And, I see splunkd.log on HF is growing. There is no change in inputs.conf or outputs.conf before restart. What could be the reason?

[adobrzeniecki]

Run  /opt/splunk/bin/splunk btool outputs list --debug


You should see that the whitelisted index list does not include _internal. It is a precedence issue.  For us the issue was because the SplunkForwarder app did not include _internal in the whitelist for indexes. Just put this in /opt/splunk/etc/system/local/outputs.conf OR /opt/splunk/etc/SplunkForwarder/local/outputs.conf

[tcpout] 
forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry)

[tiagofbmm]

Use btool to check on your inputs for splunkd.log files:

/opt/splunk/bin/splunk btool inputs list --debug | grep -B 5 log/splunk

If there is no TCP_ROUTING sending those to somewhere strange, check the /opt/splunk/var/log on the HF to check the modtime of splunkd.

More, do a tail -f on splunkd.log to check if these are being written

Finally, on your Search Head do a | tstats count where host=yourhf by index, _time

and check if something else has stopped meanwhile from that host

[tiagofbmm]

@Rob2520 please accept an answer if it solved/helped it and upvote it. Otherwise let us know how can we help further

[MuS]

Check props.conf and/or transforms.conf if there is any filtering or routing configured. I know that _audit is not effected by those settings and therefore reaches your indexer. Also these kind of configuration changes need a Splunk restart to take effect.

cheers, MuS

[Rob2520]

MuS, i don't see props or transforms related to splunkd logs.