Thanks for the info!
Well, that instance is actually a set of 'dev' apps in there that our team uses to test and develop before deploying them out. As such, that instance is just a 'dummy' instance that works as a search head, indexer and forwarder. The slowness is actually in indexing; for example say Splunk is monitoring directoryX and new files go in at 7AM, it'll probably index into Splunk much later and not immediately.
How would a rotation policy help? Does that mean say if files are older than 2 months, it'd be good to move them to an archive place, so that Splunk no longer 'sees' it and as such no longer monitors it?
If I do a simple line count of the output of splunk list monitor, I end up with 13271, so it's probably safe to say that this instance is monitoring over 10,000 files. The ulimit is set to 4096 - is that enough or should it be at least doubled or more to be close to the # of input?
... View more