I have 3 servers each with a log file. I am planning on installing a universal forwarder on each server to push the info in these files to the receiver on the main server. Currently the log files gather no more than 5MB a day. They currently aren't getting large enough to turn over and start a new log file. My thoughts were to use the batch input type to drop the file into the Splunk directory, index it, and delete it. However because these logs aren't turning over enough I am worried getting duplicate event data. Thus, I am focused on real time forwarding on each server but concerned with the amount of resources that each forwarder will consume. With this in mind, is it better to constantly run the forwarders to avoid duplicate data, or is there another way to get the log files indexed while avoiding duplicate event data?
I would monitor the file.
The forwarders are designed to use few resources, and if that is the only input for the system, then you probably will see next to nothing for resource utilization.
You could create a scheduled task in windows or a cron job in unix to start and stop the forwarder. You should not have to leave the forwarder on for long, but that will depend on the size of the file - you could run some tests.
If the monitor is configured, then it should check the source for data very soon after splunkd starts.
Thanks for the feedback. Would you know that if in order to monitor the file the forwarder needs to be running constantly or if there is a way to have it start up every so often to minimize resource utilization. I know that even if it is running constantly it uses minimal resources, I just need to give my IT guy some numbers as to the amount of memory it actually uses constantly or if the latter is an option. Thanks for your help!