Getting Data In

HF logs is missing from _internal index


I have this Heavy Forwarder apparently not sending its own _internal logs $SPLUNK_HOME/var/log/splunk/*.log to the indexers.

What I've already checked:

  1. HF is working fine, delivering data which it's set to receive and forward.

  2. HF is phoning Deployment server fine.

  3. _audit index is being indexed fine.

  4. Using $ splunk list forward-server I see it is properly set to send data only to correct indexers.

  5. The logs are being written as expected and have proper reading permissions, e.g.:
    $ ls -ltr ~/var/log/splunk/splunkd.log
    -rw------- 1 splunk splunk 12983503 Feb 26 11:43 /opt/splunk/var/log/splunk/splunkd.log

  6. Searching for _internal index into HF returns no results as supposed to be.

Any ideas about what is going on?

There is already a question about it in Answers, but not satisfying answered...



0 Karma


Do you see no logs at all? Like, no metrics, no audit data as well or just _internal?


0 Karma


Only _intental, _audit is fine.

0 Karma


you may want to check your outputs.conf for forwardedindex* and see if _internal is missing under tcpout stanza or if you have custom config like -

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...