Monitoring Splunk

Forwarder Memory

jszyba
New Member

I have 3 servers each with a log file. I am planning on installing a universal forwarder on each server to push the info in these files to the receiver on the main server. Currently the log files gather no more than 5MB a day. They currently aren't getting large enough to turn over and start a new log file. My thoughts were to use the batch input type to drop the file into the Splunk directory, index it, and delete it. However because these logs aren't turning over enough I am worried getting duplicate event data. Thus, I am focused on real time forwarding on each server but concerned with the amount of resources that each forwarder will consume. With this in mind, is it better to constantly run the forwarders to avoid duplicate data, or is there another way to get the log files indexed while avoiding duplicate event data?

0 Karma

lukejadamec
Super Champion

I would monitor the file.

The forwarders are designed to use few resources, and if that is the only input for the system, then you probably will see next to nothing for resource utilization.

lukejadamec
Super Champion

You could create a scheduled task in windows or a cron job in unix to start and stop the forwarder. You should not have to leave the forwarder on for long, but that will depend on the size of the file - you could run some tests.
If the monitor is configured, then it should check the source for data very soon after splunkd starts.

0 Karma

jszyba
New Member

Thanks for the feedback. Would you know that if in order to monitor the file the forwarder needs to be running constantly or if there is a way to have it start up every so often to minimize resource utilization. I know that even if it is running constantly it uses minimal resources, I just need to give my IT guy some numbers as to the amount of memory it actually uses constantly or if the latter is an option. Thanks for your help!

0 Karma
Get Updates on the Splunk Community!

Security Highlights: September 2022 Newsletter

 September 2022 The Splunk App for Fraud Analytics (SFA) is now Splunk SupportedUse your existing Splunk ...

Platform Highlights | September 2022 Newsletter

 September 2022 What’s New in 9.0 and How to UpgradeGet a walk through of what is new Splunk Enterprise 9.0 ...

Observability Highlights | September 2022 Newsletter

 September 2022 Splunk Observability SuiteAccess to "Classic" SignalFx Interface Will be Removed on Sept 30, ...