Getting Data In
Highlighted

Why is Splunk universal forwarder not indexing data from all log files?

Path Finder

Hello

I am running Splunk as not root user. my Splunk universal forwarder is not indexing data from all files.

when i run Splunk list monitor, it is listing all my files that i have mentioned in input stanza, but it is not indexing those files. i have checked privileges for those files and as a non root user i can cat those files.

can anyone help me??

0 Karma
Highlighted

Re: Why is Splunk universal forwarder not indexing data from all log files?

Esteemed Legend

What do you get with this search:

index=_* host=<YourHost>

If nothing, then the problem is that your host is not sending to your indexers.

0 Karma
Highlighted

Re: Why is Splunk universal forwarder not indexing data from all log files?

Splunk Employee
Splunk Employee

Yup, second stop would be to check /opt/splunkforwarder/var/log/splunk/splunkd.log directly on the forwarder host for any hints as to what may be happening.

0 Karma
Highlighted

Re: Why is Splunk universal forwarder not indexing data from all log files?

Path Finder

i have checked splunkd.log there are no errors

0 Karma
Highlighted

Re: Why is Splunk universal forwarder not indexing data from all log files?

Path Finder

it is giving me events with _internal index.

0 Karma
Highlighted

Re: Why is Splunk universal forwarder not indexing data from all log files?

Splunk Employee
Splunk Employee

Next thing to check would be whether the index you have configured for each input on the forwarder are defined on the indexing tier. Otherwise you will see error messages in internal that look like this: "received event for unconfigured/disabled/deleted index='[indexname]' with source='[source]' host='[hostname]' sourcetype='[sourcetype]' (n missing total)"

You can also run this search to find out whether the indexer is receiving any data from the forwarder:
index=internal sourcetype=splunkd group=perhostthruput series=yourforwarderhosthere | timechart sum(kb) as totalkb by series limit=0

0 Karma
Highlighted

Re: Why is Splunk universal forwarder not indexing data from all log files?

Esteemed Legend

Check to make sure that the clock on the forwarder is correct and the TZ is accounted for in the TZ setting in props.conf. It may be that the events are being "sent to the future". To check this, search for "All Time".

Also login to the server as the user that is running splunk and verify that you can read the file contents manually. If you have a permission problem manually, then this is the problem for Splunk, too.

0 Karma