Getting Data In

HF logs is missing from _internal index

tcmarquesi
Explorer

I have this Heavy Forwarder apparently not sending its own _internal logs $SPLUNK_HOME/var/log/splunk/*.log to the indexers.

What I've already checked:

  1. HF is working fine, delivering data which it's set to receive and forward.

  2. HF is phoning Deployment server fine.

  3. _audit index is being indexed fine.

  4. Using $ splunk list forward-server I see it is properly set to send data only to correct indexers.

  5. The logs are being written as expected and have proper reading permissions, e.g.:
    $ ls -ltr ~/var/log/splunk/splunkd.log
    -rw------- 1 splunk splunk 12983503 Feb 26 11:43 /opt/splunk/var/log/splunk/splunkd.log

  6. Searching for _internal index into HF returns no results as supposed to be.

Any ideas about what is going on?

There is already a question about it in Answers, but not satisfying answered...

https://answers.splunk.com/answers/686484/why-are-internal-logs-from-heavy-forwarderhf-not-g.html

Thanks,

TCM

0 Karma

skalliger
SplunkTrust
SplunkTrust

Do you see no logs at all? Like, no metrics, no audit data as well or just _internal?

Skalli

0 Karma

tcmarquesi
Explorer

Only _intental, _audit is fine.

0 Karma

lakshman239
SplunkTrust
SplunkTrust

you may want to check your outputs.conf for forwardedindex* and see if _internal is missing under tcpout stanza or if you have custom config like - https://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Forwardsearchheaddata

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) v3.54.0

The Splunk Threat Research Team (STRT) recently released Enterprise Security Content Update (ESCU) v3.54.0 and ...

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...