topic Re: Logs not coming in Getting Data In

Logs not coming

qwaszx012 — Wed, 30 Sep 2020 05:32:22 GMT

Logs are not coming to splunk enterprise. I've found below error in splunkd.log file in (../splunkforwarder/var/log/splunk/splunkd.log)

error: "05-20-2020 10:33:28.196 +0000 WARN FilesystemChangeWatcher - removed WFS_EXISTS direntname='some_log_path' stat_failure_was_temporary"

All the log paths and directories have 755 permissions recursively, but still unable to see logs.

Kindly help me on this.

Re: Logs not coming

richgalloway — Wed, 27 May 2020 12:15:41 GMT

Please share the inputs.conf settings for that forwarder.
Are the indexers configured to receive data? Have you checked the firewalls? Has this ever worked? If so, what changed?

Re: Logs not coming

qwaszx012 — Wed, 30 Sep 2020 05:32:28 GMT

Hi Rich,
Due to some privacy issues I cannot completely disclose inputs.conf information.

inputs.conf

[monitor:///some_log_path.log*]
index = some_index_name
blacklist = .(gz|zip|bkz|arch|etc)$
sourcetype = some_source_type

1)Yes, indexers are configured to receive data.
3)No, this has not worked from starting.

what could cause that above error?

Re: Logs not coming

richgalloway — Wed, 27 May 2020 13:52:29 GMT

Have you checked your firewalls? Can you connect from the UF to an indexer using a program such as telnet, curl, or traceroute? Do you see the forwarder's internal logs in the indexers?

The log message cited is a warning, which seems to indicate a previous failure condition no longer exists.

Re: Logs not coming

qwaszx012 — Wed, 30 Sep 2020 05:32:42 GMT

Hi Rich,

I could connect to both indexer as well as deployment server from the forwarder using netcat(nc) command. The forwarder we're using is a heavy forwarder.

Yes we see internal logs of forwarder.

output of index="_internal" ->

05-28-2020 03:18:23.470 +0000 WARN FilesystemChangeWatcher - removed WFS_EXISTS direntname= stat_failure_was_temporary
05-28-2020 03:26:26.373 +0000 WARN FilesystemChangeWatcher - removed WFS_EXISTS direntname=../splunkforwarder/var/log/fwdLicenseUpdate/fwdLicenseUpdate.log stat_failure_was_temporary

Re: Logs not coming

richgalloway — Thu, 28 May 2020 12:59:53 GMT

Check the permissions on the filepath that is not getting to Splunk. Verify the HF has read access to the file.
Verify the directory in question contains files that do not end with .gz, .zip, .bkz, .arch,. or .etc.

Re: Logs not coming

qwaszx012 — Fri, 29 May 2020 05:41:59 GMT

The directories have 750 permissions and log files have 640 permissions recursively. Also verified that logs files do not end with .gz, .zip, .bkz, .arch,. or .etc.

Re: Logs not coming

Richfez — Fri, 29 May 2020 12:13:26 GMT

Are these local, physically attached mount points and local directories? Or are they mounted from elsewhere (or even, mounted from elsewhere and elsewhere is this same machine?)

And what filesystem is each of these?

You could test by making a file in the simplest of places, like in /root or /home/user, and building a new input and seeing if that works. Just make a throwaway index to send that to. If that doesn't work, let us know what new or different errors you get from that.

If it does work, move that test location around a bit and see if you can find the commonality.