I'm considering usage of splunk-forwarder to integrate a system that generates many small files that contain log messages, i.e. at times more than a thousand per second. Once the files reach splunk they can be deleted.
I wonder how will the forwarded handle this situation. I've read that it can monitor well about a 100 files. Should I implement other jobs to move the processed files and how should I know if a file is processed?
The other approach that I could take is to change the system to log a rotating file. So which one do you think is better?
... View more