topic Best/optimum log file size? in Getting Data In

Best/optimum log file size?

lguinn2 — Sun, 27 Sep 2015 05:34:38 GMT

I saw a recommendation that Splunk works better with smaller log files. But what does "small" mean?

I would assume that 2GB is small, but maybe not? I can set log file rotation based on size or time, and I wanted to use size to optimize Splunk's ability to read the files...

I found a couple of other answers that touch on this subject, including one that says "there isn't really a max size except for file system limits." Okay for the maximum, but what is optimum? No one actually says "it's better if your log files don't exceed size X."

Re: Best/optimum log file size?

muebel — Mon, 28 Sep 2015 01:24:11 GMT

Was that from the "Logging: Proven operational practices" section? I couldn't find a direct match for "Splunk works better with smaller log files"

Re: Best/optimum log file size?

lguinn2 — Mon, 28 Sep 2015 20:10:50 GMT

Yes, the direct quote is "Use smaller log files."

Re: Best/optimum log file size?

masonmorales — Mon, 28 Sep 2015 20:15:04 GMT

1 GB is the recommended size that I heard at conf 2015.

Re: Best/optimum log file size?

muebel — Tue, 29 Sep 2015 11:48:17 GMT

My take on this is that it might not be that Splunk work's better per se, but that it is better to setup smaller files through rotation by the fact that, since Splunk is indexing the contents of the files, you don't have to leave as much local on the system's disk.

Re: Best/optimum log file size?

lguinn2 — Fri, 16 Oct 2015 07:19:44 GMT

Can you give more info on where you heard this? Just curious...