topic Re: Why does forwarding stop until i restart splunk in Getting Data In

Why does forwarding stop until i restart splunk

capilarity — Fri, 24 May 2013 14:13:32 GMT

I have a 4.3.3 UF on a windows 2008r2 box that was forwarding windows event logs quite happily.
It's now stopped forwarding but, if I restart splunk on the forwarding server, the missing events are forwarded, but no new events until I restart splunk again.
Short of restarting splunk every 5 minutes, can any one suggest why this might be happening?

Re: Why does forwarding stop until i restart splunk

lguinn2 — Fri, 24 May 2013 16:19:24 GMT

On the forwarder, have you looked at the splunkd log? You will find it in the subdirectory $SPLUNK_HOME\var\log\splunk

Let us know what you find there...

Re: Why does forwarding stop until i restart splunk

SarahBOA — Wed, 16 Apr 2014 15:47:42 GMT

We are experiencing a few issues with our windows forwarders and one of them sounds like it might be the same. We have this issue where we get splunk internal logs constantly, but monitored files are only sent on shutdown of the universal forwarder. This problem appears to be that the splunk forwarder was trying to "restart" too quickly. When we tried to restart the forwarder we received an error message that the process was taking too long - but it appeared stopped in the windows serverice listing so we started it up again. No error message was received on startup. We then received the internal logs as expected, but didn't receive the application log file we were monitoring. We then stopped the forwarder (received the error message again) and waited about 5 minutes. After 5 minutes we started the forwarder and both the splunk internal logs and the monitored log files were continuously coming through. It appears that the stop, pause for a longer time, then start appeared to fix this issue.

This did not fix the issue where the splunkd logs only came through when the agent was stopped and the monitored logs never came through.

Re: Why does forwarding stop until i restart splunk

capilarity — Tue, 22 Apr 2014 13:24:50 GMT

Accepted the answer by mistake - any way you can unaccept?

The issue went away when we upgraded, we are now splunk 6 and not an issue so far......

Re: Why does forwarding stop until i restart splunk

jareddjenkins — Thu, 22 Jun 2017 16:08:17 GMT

I had the same symptoms, it was a configuration issue.
Make sure you fully understand ignoreOlderThan=
https://docs.splunk.com/Documentation/Splunk/6.6.1/Admin/Inputsconf

In my case logs were not written to for 7 + days and then splunk will no longer try to read from that file even when new events appeard in the file..

Re: Why does forwarding stop until i restart splunk

jareddjenkins — Thu, 22 Jun 2017 16:11:44 GMT

In my case, it was a misunderstanding of ignoreOlderThan= in inputs.conf.

ignoreOlderThan will completely ignore files that ever reach this threshold.
From the inputs.conf documentation.
"Do NOT select a time that files you want to read could reach in age, even temporarily"

My files wouldn't write to the logs for several weeks and then begin writing again. Splunk would not even try to ingest them.