Forwarder stop reads monitored logs after restart

[robertosegantin]

Hi to all,

I have several Forwarders on Windows that monitor more than 20k items each (folder and logs inside them).
In total I'm monitoring more than 200k logs.

This is my inputs.conf deployed on those forwarders:

[monitor://c:\FirstFolder\Log\service\System\20*\PROXY\...] 
disabled = false 
index=my_index 
sourcetype=proxy 
alwaysOpenFile=1

[monitor://c:\\FirstFolder\Log\service\System\20*\00*\*.log] 
disabled = false 
index=my_index 
sourcetype=process 
alwaysOpenFile=1

[monitor://c:\\FirstFolder\Log\service\System\20*\00*.log] 
disabled = false 
index=my_index 
sourcetype=device
alwaysOpenFile=1

[monitor://c:\\FirstFolder\Log\service\System\20*\ERROR_*.log] 
disabled = false 
index=main 
sourcetype=error 
alwaysOpenFile=1

When I restart the forwarders, they does not monitor anymore the files that the forwarders read before, even if the log files will be written after the restart.

[vinod94]

Hi @robertosegantin ,

Did you try cheking splunk internal logs? Are der any ERRORS in log_level?

index=_internal

[markusspitzli]

We had similar issues with a unix forwarder. I has to monitor over a million of files. It stopped sending logfiles and had high CPU consumption. I assume you are struggling with the same issue as we did.

The UF doesnt like haveing lots of open files like you configured with alwaysOpenfile=1 Even if you get rid of this option I doubt that the UF will work.

Try using the batch://option. It reads and deletes the file. if you dont want to delete the file on your server I suggest, that you write a script in which the files are copied to a temporary directory.

In the end you will have configuration like this (inputs.conf):

[batch://c:\FirstFolder\Log\service\System\20*\PROXY\...]
disabled = false 
index=my_index 
sourcetype=proxy 
move_policy = sinkhole