Getting Data In

HF logs is missing from _internal index


I have this Heavy Forwarder apparently not sending its own _internal logs $SPLUNK_HOME/var/log/splunk/*.log to the indexers.

What I've already checked:

  1. HF is working fine, delivering data which it's set to receive and forward.

  2. HF is phoning Deployment server fine.

  3. _audit index is being indexed fine.

  4. Using $ splunk list forward-server I see it is properly set to send data only to correct indexers.

  5. The logs are being written as expected and have proper reading permissions, e.g.:
    $ ls -ltr ~/var/log/splunk/splunkd.log
    -rw------- 1 splunk splunk 12983503 Feb 26 11:43 /opt/splunk/var/log/splunk/splunkd.log

  6. Searching for _internal index into HF returns no results as supposed to be.

Any ideas about what is going on?

There is already a question about it in Answers, but not satisfying answered...



0 Karma


Do you see no logs at all? Like, no metrics, no audit data as well or just _internal?


0 Karma


Only _intental, _audit is fine.

0 Karma


you may want to check your outputs.conf for forwardedindex* and see if _internal is missing under tcpout stanza or if you have custom config like -

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...