Getting Data In

Why is Splunk universal forwarder not indexing data from all log files?

AzmathShaik
Path Finder

Hello

I am running Splunk as not root user. my Splunk universal forwarder is not indexing data from all files.

when i run Splunk list monitor, it is listing all my files that i have mentioned in input stanza, but it is not indexing those files. i have checked privileges for those files and as a non root user i can cat those files.

can anyone help me??

0 Karma

woodcock
Esteemed Legend

Check to make sure that the clock on the forwarder is correct and the TZ is accounted for in the TZ setting in props.conf. It may be that the events are being "sent to the future". To check this, search for "All Time".

Also login to the server as the user that is running splunk and verify that you can read the file contents manually. If you have a permission problem manually, then this is the problem for Splunk, too.

0 Karma

woodcock
Esteemed Legend

What do you get with this search:

index=_* host=<YourHost>

If nothing, then the problem is that your host is not sending to your indexers.

0 Karma

AzmathShaik
Path Finder

it is giving me events with _internal index.

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

Next thing to check would be whether the index you have configured for each input on the forwarder are defined on the indexing tier. Otherwise you will see error messages in _internal that look like this: "received event for unconfigured/disabled/deleted index='[index_name]' with source='[source]' host='[hostname]' sourcetype='[sourcetype]' (n missing total)"

You can also run this search to find out whether the indexer is receiving any data from the forwarder:
index=_internal sourcetype=splunkd group=per_host_thruput series=your_forwarder_host_here | timechart sum(kb) as totalkb by series limit=0

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

Yup, second stop would be to check /opt/splunkforwarder/var/log/splunk/splunkd.log directly on the forwarder host for any hints as to what may be happening.

0 Karma

AzmathShaik
Path Finder

i have checked splunkd.log there are no errors

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...