https://community.splunk.com/t5/Knowledge-Management/Spunk-forwarder-scalability/m-p/202746#M6908 <P>I'm considering usage of splunk-forwarder to integrate a system that generates many small files that contain log messages, i.e. at times more than a thousand per second. Once the files reach splunk they can be deleted.</P> <P>I wonder how will the forwarded handle this situation. I've read that it can monitor well about a 100 files. Should I implement other jobs to move the processed files and how should I know if a file is processed?</P> <P>The other approach that I could take is to change the system to log a rotating file. So which one do you think is better?</P> Wed, 21 Sep 2016 07:33:45 GMT dimitarvalov 2016-09-21T07:33:45Z https://community.splunk.com/t5/Knowledge-Management/Spunk-forwarder-scalability/m-p/202746#M6908 <P>I'm considering usage of splunk-forwarder to integrate a system that generates many small files that contain log messages, i.e. at times more than a thousand per second. Once the files reach splunk they can be deleted.</P> <P>I wonder how will the forwarded handle this situation. I've read that it can monitor well about a 100 files. Should I implement other jobs to move the processed files and how should I know if a file is processed?</P> <P>The other approach that I could take is to change the system to log a rotating file. So which one do you think is better?</P> Wed, 21 Sep 2016 07:33:45 GMT https://community.splunk.com/t5/Knowledge-Management/Spunk-forwarder-scalability/m-p/202746#M6908 dimitarvalov 2016-09-21T07:33:45Z https://community.splunk.com/t5/Knowledge-Management/Spunk-forwarder-scalability/m-p/202747#M6909 <P>I would say prefer to log to a rotating file. I have experience from Splunking an application that was producing several thousand files an hour and it was not pretty. Monitoring that amount of files will result in a performance hit which is disproportionate to the amount of data to be ingested.</P> <P>Therefore: prefer fewer log files.</P> Wed, 21 Sep 2016 13:37:45 GMT https://community.splunk.com/t5/Knowledge-Management/Spunk-forwarder-scalability/m-p/202747#M6909 echalex 2016-09-21T13:37:45Z