Monitoring Splunk

Why are _internal logs from heavy forwarder(HF) not getting to indexers after a Splunkd restart but _audit are?

Rob2520
Communicator

All of a sudden, _internal logs from HF stopped coming to indexers after a Splunkd restart. But, i see _audit logs making it to indexers. And, I see splunkd.log on HF is growing. There is no change in inputs.conf or outputs.conf before restart. What could be the reason?

0 Karma

adobrzeniecki
Path Finder

Run  /opt/splunk/bin/splunk btool outputs list --debug


You should see that the whitelisted index list does not include _internal. It is a precedence issue.  For us the issue was because the SplunkForwarder app did not include _internal in the whitelist for indexes. Just put this in /opt/splunk/etc/system/local/outputs.conf OR /opt/splunk/etc/SplunkForwarder/local/outputs.conf

[tcpout] 
forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry)

 

 

0 Karma

tiagofbmm
Influencer

Use btool to check on your inputs for splunkd.log files:

/opt/splunk/bin/splunk btool inputs list --debug | grep -B 5 log/splunk

If there is no TCP_ROUTING sending those to somewhere strange, check the /opt/splunk/var/log on the HF to check the modtime of splunkd.

More, do a tail -f on splunkd.log to check if these are being written

Finally, on your Search Head do a | tstats count where host=yourhf by index, _time

and check if something else has stopped meanwhile from that host

0 Karma

tiagofbmm
Influencer

@Rob2520 please accept an answer if it solved/helped it and upvote it. Otherwise let us know how can we help further

0 Karma

MuS
Legend

Check props.conf and/or transforms.conf if there is any filtering or routing configured. I know that _audit is not effected by those settings and therefore reaches your indexer. Also these kind of configuration changes need a Splunk restart to take effect.

cheers, MuS

0 Karma

Rob2520
Communicator

MuS, i don't see props or transforms related to splunkd logs.

0 Karma
Get Updates on the Splunk Community!

Why You Can't Miss .conf25: Unleashing the Power of Agentic AI with Splunk & Cisco

The Defining Technology Movement of Our Lifetime The advent of agentic AI is arguably the defining technology ...

Deep Dive into Federated Analytics: Unlocking the Full Power of Your Security Data

In today’s complex digital landscape, security teams face increasing pressure to protect sprawling data across ...

Your summer travels continue with new course releases

Summer in the Northern hemisphere is in full swing, and is often a time to travel and explore. If your summer ...