Getting Data In

Splunk Forwarder Tail Fails

OldManEd
Builder

I have several forwarders that are release 4.3.2. The issue is that the log files they are configured to send to my indexers periodically rotate. For some reason Splunk will not send any more data when this happens. But, if I restart the forwarder, data starts coming in with no problems. The strange thing is that this only happens on some of the forwarders.

File permissions were checked and found to be OK.

The related errors in the splunkd.log file are;

06-14-2014 08:00:08.608 -0600 ERROR TailingProcessor - Ignoring path due to: failed to open for checksum: '/var/tmp/xxx/xxx.log' (No such file or directory)

06-14-2014 08:00:09.911 -0600 ERROR TailingProcessor - Unable to resolve path for symlink: /var/tmp/xxx/xxx.log.

Any ideas?

1 Solution

the_wolverine
Champion

Splunk bugs go unreported (or undetected) often. I suggest upgrading to a later version of the forwarder, at least version 4.3.4 or 5.0.7 to see if you still encounter the issue.

View solution in original post

scc00
Contributor

So I had the same issue. The fix was upping the permissions to the symlink. So check to see if the user for your splunk install can get to the full location and increase the permissions accordingly.

0 Karma

kamaljagga
Path Finder

I am facing the same issue and my forwarder version is 4.2.6 and forwarder runs as root. Thinking of doing a clean re-install of forwarder.
If anyone found any other solution. Kindly advise.

the_wolverine
Champion

Splunk bugs go unreported (or undetected) often. I suggest upgrading to a later version of the forwarder, at least version 4.3.4 or 5.0.7 to see if you still encounter the issue.

Kawtar
Path Finder

I have the same problem, but the version is not the problem, how can we resolve that plz ?

0 Karma

abonuccelli_spl
Splunk Employee
Splunk Employee

06-14-2014 08:00:09.911 -0600 ERROR
TailingProcessor - Unable to resolve
path for symlink:
/var/tmp/xxx/xxx.log.

can you actually resolve that symlink using ls?

tail /path/to/symlink

0 Karma

OldManEd
Builder

Yes. The link is "/var/tmp/xxx/xxx.log -> /usr/xxx/yyy/AAA/work/xxx_aaa_99.log". From what I understand, the target of the link, in this case "/usr/xxx/yyy/AAA/work/xxx_aaa_99.log", is somehow "processed" and there is a period of about 1.0 to 1.5 seconds where it doesn't exist. But once the process is complete, everything functions correctly except the forwarder doesn't continue to tail the new file.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...