Hello
I am running Splunk as not root user. my Splunk universal forwarder is not indexing data from all files.
when i run Splunk list monitor, it is listing all my files that i have mentioned in input stanza, but it is not indexing those files. i have checked privileges for those files and as a non root user i can cat those files.
can anyone help me??
Check to make sure that the clock on the forwarder is correct and the TZ is accounted for in the TZ setting in props.conf
. It may be that the events are being "sent to the future". To check this, search for "All Time".
Also login to the server as the user that is running splunk and verify that you can read the file contents manually. If you have a permission problem manually, then this is the problem for Splunk, too.
What do you get with this search:
index=_* host=<YourHost>
If nothing, then the problem is that your host is not sending to your indexers.
it is giving me events with _internal index.
Next thing to check would be whether the index you have configured for each input on the forwarder are defined on the indexing tier. Otherwise you will see error messages in _internal that look like this: "received event for unconfigured/disabled/deleted index='[index_name]' with source='[source]' host='[hostname]' sourcetype='[sourcetype]' (n missing total)"
You can also run this search to find out whether the indexer is receiving any data from the forwarder:
index=_internal sourcetype=splunkd group=per_host_thruput series=your_forwarder_host_here | timechart sum(kb) as totalkb by series limit=0
Yup, second stop would be to check /opt/splunkforwarder/var/log/splunk/splunkd.log directly on the forwarder host for any hints as to what may be happening.
i have checked splunkd.log there are no errors