topic Re: HF logs is missing from _internal index in Getting Data In

HF logs is missing from _internal index

tcmarquesi — Tue, 29 Sep 2020 23:26:00 GMT

I have this Heavy Forwarder apparently not sending its own _internal logs $SPLUNK_HOME/var/log/splunk/*.log to the indexers.

What I've already checked:

HF is working fine, delivering data which it's set to receive and forward.
HF is phoning Deployment server fine.
_audit index is being indexed fine.
Using $ splunk list forward-server I see it is properly set to send data only to correct indexers.
The logs are being written as expected and have proper reading permissions, e.g.:
$ ls -ltr ~/var/log/splunk/splunkd.log -rw------- 1 splunk splunk 12983503 Feb 26 11:43 /opt/splunk/var/log/splunk/splunkd.log
Searching for _internal index into HF returns no results as supposed to be.

Any ideas about what is going on?

There is already a question about it in Answers, but not satisfying answered...

Thanks,

TCM

skalliger — Tue, 26 Feb 2019 12:10:34 GMT

Do you see no logs at all? Like, no metrics, no audit data as well or just _internal?

Skalli

tcmarquesi — Tue, 26 Feb 2019 12:14:11 GMT

Only _intental, _audit is fine.

lakshman239 — Tue, 26 Feb 2019 12:19:39 GMT

you may want to check your outputs.conf for forwardedindex* and see if _internal is missing under tcpout stanza or if you have custom config like - https://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Forwardsearchheaddata