Getting Data In

UF is not flavour for monitoring over 10k of files?

Explorer

Hi,

I guess I'm not alone for this issue.
Any of you encountered high CPU using when UF is monitoring like over 10k of files?
In fact each file is very small. But they're required to be collected.
As I know UF would have a full list of files in memory, seems traversing the file list would spend a lot of CPU time.
This is still the same if we specified ignoreOlderThan.
And I can't reorganize customer's files

Now I'm considering to write a scheduled script to add file by file through the script, e.g. using "add oneshot".
But that's pain to keep track whether files have been captured or not.

Kindly want to listen if any other smarter suggestions.

THANKS!!

0 Karma

Ultra Champion

@philip_w - keep in mind that when the forwarder comes up, it has to build this list which is costly. The moment the original scan is over, the forwarder should be stable and consume less cpu. So, I suggest that in your testing, allow time to reach the stabilized period...

0 Karma

Explorer

Badly, it went up too high when it's kind of stabilized (1.8 core) which impacted customer's business or even consumed more resource than their business application.

0 Karma

Super Champion
0 Karma

Explorer

I meant monitoring 10k. In fact, we just need to index once since all the files are XMLs, they won't update.
As said, I can't rotate or reorganize customer's files. They're there for other business reason.

From this post, it seems setting ulimit -n to unlimited may not be the best. Currently we use ulimited. Let me check if smaller number works.
https://www.splunk.com/blog/2011/11/21/whats-your-ulimit.html

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!