Getting Data In

HF logs is missing from _internal index

tcmarquesi
Explorer

I have this Heavy Forwarder apparently not sending its own _internal logs $SPLUNK_HOME/var/log/splunk/*.log to the indexers.

What I've already checked:

  1. HF is working fine, delivering data which it's set to receive and forward.

  2. HF is phoning Deployment server fine.

  3. _audit index is being indexed fine.

  4. Using $ splunk list forward-server I see it is properly set to send data only to correct indexers.

  5. The logs are being written as expected and have proper reading permissions, e.g.:
    $ ls -ltr ~/var/log/splunk/splunkd.log
    -rw------- 1 splunk splunk 12983503 Feb 26 11:43 /opt/splunk/var/log/splunk/splunkd.log

  6. Searching for _internal index into HF returns no results as supposed to be.

Any ideas about what is going on?

There is already a question about it in Answers, but not satisfying answered...

https://answers.splunk.com/answers/686484/why-are-internal-logs-from-heavy-forwarderhf-not-g.html

Thanks,

TCM

0 Karma

skalliger
SplunkTrust
SplunkTrust

Do you see no logs at all? Like, no metrics, no audit data as well or just _internal?

Skalli

0 Karma

tcmarquesi
Explorer

Only _intental, _audit is fine.

0 Karma

lakshman239
SplunkTrust
SplunkTrust

you may want to check your outputs.conf for forwardedindex* and see if _internal is missing under tcpout stanza or if you have custom config like - https://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Forwardsearchheaddata

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...