Hi to all,
I have several Forwarders on Windows that monitor more than 20k items each (folder and logs inside them).
In total I'm monitoring more than 200k logs.
This is my inputs.conf deployed on those forwarders:
[monitor://c:\FirstFolder\Log\service\System\20*\PROXY\...] disabled = false index=my_index sourcetype=proxy alwaysOpenFile=1 [monitor://c:\\FirstFolder\Log\service\System\20*\00*\*.log] disabled = false index=my_index sourcetype=process alwaysOpenFile=1 [monitor://c:\\FirstFolder\Log\service\System\20*\00*.log] disabled = false index=my_index sourcetype=device alwaysOpenFile=1 [monitor://c:\\FirstFolder\Log\service\System\20*\ERROR_*.log] disabled = false index=main sourcetype=error alwaysOpenFile=1
When I restart the forwarders, they does not monitor anymore the files that the forwarders read before, even if the log files will be written after the restart.
We had similar issues with a unix forwarder. I has to monitor over a million of files. It stopped sending logfiles and had high CPU consumption. I assume you are struggling with the same issue as we did.
The UF doesnt like haveing lots of open files like you configured with
alwaysOpenfile=1 Even if you get rid of this option I doubt that the UF will work.
Try using the
batch://option. It reads and deletes the file. if you dont want to delete the file on your server I suggest, that you write a script in which the files are copied to a temporary directory.
In the end you will have configuration like this (inputs.conf):
[batch://c:\FirstFolder\Log\service\System\20*\PROXY\...] disabled = false index=my_index sourcetype=proxy move_policy = sinkhole