Splunk Forwarder Tail Fails

[OldManEd]

I have several forwarders that are release 4.3.2. The issue is that the log files they are configured to send to my indexers periodically rotate. For some reason Splunk will not send any more data when this happens. But, if I restart the forwarder, data starts coming in with no problems. The strange thing is that this only happens on some of the forwarders.

File permissions were checked and found to be OK.

The related errors in the splunkd.log file are;

06-14-2014 08:00:08.608 -0600 ERROR TailingProcessor - Ignoring path due to: failed to open for checksum: '/var/tmp/xxx/xxx.log' (No such file or directory)

06-14-2014 08:00:09.911 -0600 ERROR TailingProcessor - Unable to resolve path for symlink: /var/tmp/xxx/xxx.log.

Any ideas?

[scc00]

So I had the same issue. The fix was upping the permissions to the symlink. So check to see if the user for your splunk install can get to the full location and increase the permissions accordingly.

[kamaljagga]

I am facing the same issue and my forwarder version is 4.2.6 and forwarder runs as root. Thinking of doing a clean re-install of forwarder.
If anyone found any other solution. Kindly advise.

[the_wolverine · Accepted solution]

Splunk bugs go unreported (or undetected) often. I suggest upgrading to a later version of the forwarder, at least version 4.3.4 or 5.0.7 to see if you still encounter the issue.

[Kawtar]

I have the same problem, but the version is not the problem, how can we resolve that plz ?

[abonuccelli_spl]

06-14-2014 08:00:09.911 -0600 ERROR
TailingProcessor - Unable to resolve
path for symlink:
/var/tmp/xxx/xxx.log.

can you actually resolve that symlink using ls?

tail /path/to/symlink

[OldManEd]

Yes. The link is "/var/tmp/xxx/xxx.log -> /usr/xxx/yyy/AAA/work/xxx_aaa_99.log". From what I understand, the target of the link, in this case "/usr/xxx/yyy/AAA/work/xxx_aaa_99.log", is somehow "processed" and there is a period of about 1.0 to 1.5 seconds where it doesn't exist. But once the process is complete, everything functions correctly except the forwarder doesn't continue to tail the new file.