Getting Data In

Can the universal/heavy forwarder monitor a folder that is receiving a thousands of files every 15 mins?


Hi, we have our use case here that either we'll be monitoring an approximate of 6 thousand files that are updating at random interval or monitoring a folder that will receive 6 thousand files per 15 minutes that has retention period of 3 months. License-wise, the latter case is the good option but I'm worried about its performance.

We are planning on either using universal or heavy forwarder for this. Will the heavy/universal forwarder's system requirement specified in Splunk Docs be enough in this case? Will adjusting the ulimits enough to monitor a folder in the latter case?

Thank you and have a nice day!

0 Karma

Path Finder
0 Karma

Ultra Champion

Hi @rajyah

If the folder structure for the 6000 files is complex, you should do everything in your control to make the monitor stanzas as specific as possible.
Using wildcard monitor statements over deep file systems has a significant performance impact, so if this can be avoided it would be of benefit.

As long as the box is sufficiently resourced (Network/Memory/IO) I don't think you have too much too worry about - the Splunk recommended ulimit is 64k.

Personally, I think I would opt for option 1 (files updated at random), as this (presumably) would stagger the changes throughout an arbitrary 15 minute period, versus one big change every quarter of an hour. - I also don't understand your reference to licencing.

Your biggest challenge will be making sure your indexing pipelines are big enough to keep up with the rate of change, though you have not mentioned anything about volume.

If my comment helps, please give it a thumbs up!
0 Karma
Get Updates on the Splunk Community!

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...