Monitoring Splunk

Forwarder Memory

jszyba
New Member

I have 3 servers each with a log file. I am planning on installing a universal forwarder on each server to push the info in these files to the receiver on the main server. Currently the log files gather no more than 5MB a day. They currently aren't getting large enough to turn over and start a new log file. My thoughts were to use the batch input type to drop the file into the Splunk directory, index it, and delete it. However because these logs aren't turning over enough I am worried getting duplicate event data. Thus, I am focused on real time forwarding on each server but concerned with the amount of resources that each forwarder will consume. With this in mind, is it better to constantly run the forwarders to avoid duplicate data, or is there another way to get the log files indexed while avoiding duplicate event data?

0 Karma

lukejadamec
Super Champion

I would monitor the file.

The forwarders are designed to use few resources, and if that is the only input for the system, then you probably will see next to nothing for resource utilization.

lukejadamec
Super Champion

You could create a scheduled task in windows or a cron job in unix to start and stop the forwarder. You should not have to leave the forwarder on for long, but that will depend on the size of the file - you could run some tests.
If the monitor is configured, then it should check the source for data very soon after splunkd starts.

0 Karma

jszyba
New Member

Thanks for the feedback. Would you know that if in order to monitor the file the forwarder needs to be running constantly or if there is a way to have it start up every so often to minimize resource utilization. I know that even if it is running constantly it uses minimal resources, I just need to give my IT guy some numbers as to the amount of memory it actually uses constantly or if the latter is an option. Thanks for your help!

0 Karma
Get Updates on the Splunk Community!

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...

Thank You for Celebrating CX Day with Splunk!

Yesterday the entire team at Splunk + Cisco joined the global celebration of CX Day - celebrating our ...