topic Re: Splunk Universal Forwarder Error logs in Splunk Enterprise

Are there any logs maintained by the Splunk Universal forwarder in case of log processing failures?

subramanianers — Fri, 11 Feb 2022 17:15:34 GMT

Are there any logs maintained by the Splunk Universal forwarder in case of log processing failures? I would like to setup an alert and dashboard for the log processing failures that occur while the Universal forwarder tries logs to my indexer. I need this for compliance purposes.

I checked out the URL - https://docs.splunk.com/Documentation/Splunk/8.2.4/Troubleshooting/WhatSplunklogsaboutitself, and thought splunkd_stderr.log on my servers would do the trick. But when I checked the corresponding logs, I could see only start and stop messages. Can someone provide me the keyword search and the filename for log processing failures in the Splunk Universal Forwarder?

Re: Splunk Universal Forwarder Error logs

PickleRick — Fri, 11 Feb 2022 07:36:21 GMT

Logs forwarded to _internal can provide some insight into input processing errors or stuff like event breaking but be wary that in case something really breaks down and the forwarder crashes or stops forwarding... you're not getting logs to _internal either. After all, they are the logs from /opt/splunk/var/log/splunk/* forwarded to indexer by the UF itself. So if the UF stops forwarding it only logs to a local file but doesn't send the events from that file to the upstream indexers.

Re: Splunk Universal Forwarder Error logs

subramanianers — Fri, 11 Feb 2022 07:39:03 GMT

Hi @PickleRick . Thanks for replying. I understand that if something is truly broken in the UF, then logs will not be sent to Splunk. I am worried only about the processing errors or invalidations that might have happen in UF, not the case of UF breaking down completely. Is there a particular log file or query that you could point me to so that I can achieve this use case?

Re: Splunk Universal Forwarder Error logs

PickleRick — Fri, 11 Feb 2022 07:42:01 GMT

Usualy the warnings get logged simply into splunkd.log.

In case of scripted/modular inputs they can however have their own separate log files but typical splunk components log to splunkd.log.