Spunk forwarder scalability

[dimitarvalov]

I'm considering usage of splunk-forwarder to integrate a system that generates many small files that contain log messages, i.e. at times more than a thousand per second. Once the files reach splunk they can be deleted.

I wonder how will the forwarded handle this situation. I've read that it can monitor well about a 100 files. Should I implement other jobs to move the processed files and how should I know if a file is processed?

The other approach that I could take is to change the system to log a rotating file. So which one do you think is better?

[echalex · Accepted solution]

I would say prefer to log to a rotating file. I have experience from Splunking an application that was producing several thousand files an hour and it was not pretty. Monitoring that amount of files will result in a performance hit which is disproportionate to the amount of data to be ingested.

Therefore: prefer fewer log files.