All of a sudden, _internal logs from HF stopped coming to indexers after a Splunkd restart. But, i see _audit logs making it to indexers. And, I see splunkd.log on HF is growing. There is no change in inputs.conf or outputs.conf before restart. What could be the reason?
Use btool to check on your inputs for splunkd.log files:
/opt/splunk/bin/splunk btool inputs list --debug | grep -B 5 log/splunk
If there is no TCP_ROUTING sending those to somewhere strange, check the /opt/splunk/var/log on the HF to check the modtime of splunkd.
More, do a tail -f on splunkd.log to check if these are being written
Finally, on your Search Head do a
| tstats count where host=yourhf by index, _time
and check if something else has stopped meanwhile from that host
transforms.conf if there is any filtering or routing configured. I know that
_audit is not effected by those settings and therefore reaches your indexer. Also these kind of configuration changes need a Splunk restart to take effect.