Monitoring Splunk

Forwarder Memory

jszyba
New Member

I have 3 servers each with a log file. I am planning on installing a universal forwarder on each server to push the info in these files to the receiver on the main server. Currently the log files gather no more than 5MB a day. They currently aren't getting large enough to turn over and start a new log file. My thoughts were to use the batch input type to drop the file into the Splunk directory, index it, and delete it. However because these logs aren't turning over enough I am worried getting duplicate event data. Thus, I am focused on real time forwarding on each server but concerned with the amount of resources that each forwarder will consume. With this in mind, is it better to constantly run the forwarders to avoid duplicate data, or is there another way to get the log files indexed while avoiding duplicate event data?

0 Karma

lukejadamec
Super Champion

I would monitor the file.

The forwarders are designed to use few resources, and if that is the only input for the system, then you probably will see next to nothing for resource utilization.

lukejadamec
Super Champion

You could create a scheduled task in windows or a cron job in unix to start and stop the forwarder. You should not have to leave the forwarder on for long, but that will depend on the size of the file - you could run some tests.
If the monitor is configured, then it should check the source for data very soon after splunkd starts.

0 Karma

jszyba
New Member

Thanks for the feedback. Would you know that if in order to monitor the file the forwarder needs to be running constantly or if there is a way to have it start up every so often to minimize resource utilization. I know that even if it is running constantly it uses minimal resources, I just need to give my IT guy some numbers as to the amount of memory it actually uses constantly or if the latter is an option. Thanks for your help!

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...