Logs are not coming to splunk enterprise. I've found below error in splunkd.log file in (../splunkforwarder/var/log/splunk/splunkd.log)
error: "05-20-2020 10:33:28.196 +0000 WARN FilesystemChangeWatcher - removed WFS_EXISTS direntname='some_log_path' stat_failure_was_temporary"
All the log paths and directories have 755 permissions recursively, but still unable to see logs.
Kindly help me on this.
Are these local, physically attached mount points and local directories? Or are they mounted from elsewhere (or even, mounted from elsewhere and elsewhere is this same machine?)
And what filesystem is each of these?
You could test by making a file in the simplest of places, like in /root or /home/user, and building a new input and seeing if that works. Just make a throwaway index to send that to. If that doesn't work, let us know what new or different errors you get from that.
If it does work, move that test location around a bit and see if you can find the commonality.
Please share the inputs.conf settings for that forwarder.
Are the indexers configured to receive data? Have you checked the firewalls? Has this ever worked? If so, what changed?
Due to some privacy issues I cannot completely disclose inputs.conf information.
index = some_index_name
blacklist = .(gz|zip|bkz|arch|etc)$
sourcetype = some_source_type
1)Yes, indexers are configured to receive data.
3)No, this has not worked from starting.
what could cause that above error?
Have you checked your firewalls? Can you connect from the UF to an indexer using a program such as telnet, curl, or traceroute? Do you see the forwarder's internal logs in the indexers?
The log message cited is a warning, which seems to indicate a previous failure condition no longer exists.
I could connect to both indexer as well as deployment server from the forwarder using netcat(nc) command. The forwarder we're using is a heavy forwarder.
Yes we see internal logs of forwarder.
output of index="_internal" ->
05-28-2020 03:18:23.470 +0000 WARN FilesystemChangeWatcher - removed WFS_EXISTS direntname= stat_failure_was_temporary
05-28-2020 03:26:26.373 +0000 WARN FilesystemChangeWatcher - removed WFS_EXISTS direntname=../splunkforwarder/var/log/fwdLicenseUpdate/fwdLicenseUpdate.log stat_failure_was_temporary
Check the permissions on the filepath that is not getting to Splunk. Verify the HF has read access to the file.
Verify the directory in question contains files that do not end with .gz, .zip, .bkz, .arch,. or .etc.