https://community.splunk.com/t5/Getting-Data-In/Can-the-universal-heavy-forwarder-monitor-a-folder-that-is/m-p/431793#M75552 <P>Hi @rajyah </P> <P>If the folder structure for the 6000 files is complex, you should do everything in your control to make the monitor stanzas as specific as possible. <BR /> Using wildcard monitor statements over deep file systems has a significant performance impact, so if this can be avoided it would be of benefit.</P> <P>As long as the box is sufficiently resourced (Network/Memory/IO) I don't think you have too much too worry about - the Splunk recommended ulimit is 64k.</P> <P>Personally, I think I would opt for option 1 (files updated at random), as this (presumably) would stagger the changes throughout an arbitrary 15 minute period, versus one big change every quarter of an hour. - I also don't understand your reference to licencing.</P> <P>Your biggest challenge will be making sure your indexing pipelines are big enough to keep up with the rate of change, though you have not mentioned anything about volume.</P> Thu, 14 Mar 2019 09:45:44 GMT nickhills 2019-03-14T09:45:44Z https://community.splunk.com/t5/Getting-Data-In/Can-the-universal-heavy-forwarder-monitor-a-folder-that-is/m-p/431792#M75551 <P>Hi, we have our use case here that either we'll be monitoring an approximate of 6 thousand files that are updating at random interval or monitoring a folder that will receive 6 thousand files per 15 minutes that has retention period of 3 months. License-wise, the latter case is the good option but I'm worried about its performance.</P> <P>We are planning on either using universal or heavy forwarder for this. Will the heavy/universal forwarder's system requirement specified in Splunk Docs be enough in this case? Will adjusting the ulimits enough to monitor a folder in the latter case?</P> <P>Thank you and have a nice day!</P> Thu, 14 Mar 2019 07:38:14 GMT https://community.splunk.com/t5/Getting-Data-In/Can-the-universal-heavy-forwarder-monitor-a-folder-that-is/m-p/431792#M75551 rajyah 2019-03-14T07:38:14Z https://community.splunk.com/t5/Getting-Data-In/Can-the-universal-heavy-forwarder-monitor-a-folder-that-is/m-p/431793#M75552 <P>Hi @rajyah </P> <P>If the folder structure for the 6000 files is complex, you should do everything in your control to make the monitor stanzas as specific as possible. <BR /> Using wildcard monitor statements over deep file systems has a significant performance impact, so if this can be avoided it would be of benefit.</P> <P>As long as the box is sufficiently resourced (Network/Memory/IO) I don't think you have too much too worry about - the Splunk recommended ulimit is 64k.</P> <P>Personally, I think I would opt for option 1 (files updated at random), as this (presumably) would stagger the changes throughout an arbitrary 15 minute period, versus one big change every quarter of an hour. - I also don't understand your reference to licencing.</P> <P>Your biggest challenge will be making sure your indexing pipelines are big enough to keep up with the rate of change, though you have not mentioned anything about volume.</P> Thu, 14 Mar 2019 09:45:44 GMT https://community.splunk.com/t5/Getting-Data-In/Can-the-universal-heavy-forwarder-monitor-a-folder-that-is/m-p/431793#M75552 nickhills 2019-03-14T09:45:44Z https://community.splunk.com/t5/Getting-Data-In/Can-the-universal-heavy-forwarder-monitor-a-folder-that-is/m-p/431794#M75553 <P>@nickhillscpl</P> <P>Any comments on my question: <BR /> <A href="https://answers.splunk.com/answers/738011/when-universal-forwarder-using-wildcard-monitor-st.html">https://answers.splunk.com/answers/738011/when-universal-forwarder-using-wildcard-monitor-st.html</A> </P> <P>Thanks a lot</P> Mon, 08 Apr 2019 17:51:44 GMT https://community.splunk.com/t5/Getting-Data-In/Can-the-universal-heavy-forwarder-monitor-a-folder-that-is/m-p/431794#M75553 imgarytan 2019-04-08T17:51:44Z