Getting Data In

Why am I seeing data from a heavy forwarder, but nothing in _internal index?

rfitch
Path Finder

I'm seeing some curious behavior our of two of my heavy forwarders. They aren't reporting data into _internal, but I am seeing app data from things I have installed on them. I checked out the logs on the heavy forwarders and the only errors I see are something about a django secret key missing "The SECRET_KEY setting must not be empty " in the web_service.log

Any idea what would cause this behavior?

0 Karma
1 Solution

saravanan90
Contributor

Splunk Heavy forwarder will not forward the _internal data by its own. We can add the below to make this forwarding to indexers.

[tcpout]
forwardedindex.3.whitelist = (_internal)

If you have two or more index, please mention this with pipe.
forwardedindex.3.whitelist = (_internal|_audit)

View solution in original post

saravanan90
Contributor

Splunk Heavy forwarder will not forward the _internal data by its own. We can add the below to make this forwarding to indexers.

[tcpout]
forwardedindex.3.whitelist = (_internal)

If you have two or more index, please mention this with pipe.
forwardedindex.3.whitelist = (_internal|_audit)

sloshburch
Splunk Employee
Splunk Employee

If I understand correctly, the forwarder is sending data, but not from select indexes (in this case _internal). That typically means the forwarder is doing it's default behavior and still needs to be configured for sending the _* data to the indexers.

The documentation at Best practice: Forward search head data to the indexer layer captures the best way to set this up. As with other global config, you'll want your Deployment Server to distribute this configuration to all your endpoints that forward data (not indexers).

somesoni2
SplunkTrust
SplunkTrust

Run btool command on your heavy forwarder to see status and configuration of your internal logs.

/opt/splunk/bin/splunk btool inputs list --debug
0 Karma

adonio
Ultra Champion

wild guess, check how much available disk you have on the drive the HF is installed on

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...