topic Re: Forwarder stop reads monitored logs after restart in Getting Data In

Forwarder stop reads monitored logs after restart

robertosegantin — Fri, 26 Oct 2018 13:41:05 GMT

Hi to all,

I have several Forwarders on Windows that monitor more than 20k items each (folder and logs inside them).
In total I'm monitoring more than 200k logs.

This is my inputs.conf deployed on those forwarders:

[monitor://c:\FirstFolder\Log\service\System\20*\PROXY\...] 
disabled = false 
index=my_index 
sourcetype=proxy 
alwaysOpenFile=1

[monitor://c:\\FirstFolder\Log\service\System\20*\00*\*.log] 
disabled = false 
index=my_index 
sourcetype=process 
alwaysOpenFile=1

[monitor://c:\\FirstFolder\Log\service\System\20*\00*.log] 
disabled = false 
index=my_index 
sourcetype=device
alwaysOpenFile=1

[monitor://c:\\FirstFolder\Log\service\System\20*\ERROR_*.log] 
disabled = false 
index=main 
sourcetype=error 
alwaysOpenFile=1

When I restart the forwarders, they does not monitor anymore the files that the forwarders read before, even if the log files will be written after the restart.

Re: Forwarder stop reads monitored logs after restart

markusspitzli — Fri, 22 Feb 2019 13:47:08 GMT

We had similar issues with a unix forwarder. I has to monitor over a million of files. It stopped sending logfiles and had high CPU consumption. I assume you are struggling with the same issue as we did.

The UF doesnt like haveing lots of open files like you configured with alwaysOpenfile=1 Even if you get rid of this option I doubt that the UF will work.

Try using the batch://option. It reads and deletes the file. if you dont want to delete the file on your server I suggest, that you write a script in which the files are copied to a temporary directory.

In the end you will have configuration like this (inputs.conf):

[batch://c:\FirstFolder\Log\service\System\20*\PROXY\...]
disabled = false 
index=my_index 
sourcetype=proxy 
move_policy = sinkhole

Re: Forwarder stop reads monitored logs after restart

vinod94 — Fri, 22 Feb 2019 15:38:53 GMT

Hi @robertosegantin ,

Did you try cheking splunk internal logs? Are der any ERRORS in log_level?

index=_internal