Getting Data In
Highlighted

How much splunk memory use should I expect while monitoring a directory with many entries?

Path Finder

If I remove the only input stanza I have on a forwarder and restart Splunk the memory usage is 2GB. How can a forwarder use that much memory if I don't have any input stanzas?

On this page it's described how Splunk use CRC checksums to keep track of which files it has seen before.

http://docs.splunk.com/Documentation/Splunk/latest/Data/Howlogfilerotationishandled

How much space does this data use on disk and in memory if you have let Splunk monitor a directory that over the years have contained more than 1 000 000 files?

This question is related to this question:
http://splunk-base.splunk.com/answers/32528/lots-of-log-files-how-can-i-reduce-forwarder-memory-usag...

Tags (1)
Highlighted

Re: How much splunk memory use should I expect while monitoring a directory with many entries?

Splunk Employee
Splunk Employee

Hi andyw. A few questions for you :

- How do you measure the memory consumption of the forwarder? Which metric (Vsize? RSS?) shows the 2Gb figure?

- How large is the directory $SPLUNKHOME/var/lib/splunk/fishbucket on the forwarder?
- How many records exist in your fishbucket? To find out, run :

$SPLUNK
HOME/bin/splunk cmd btprobe -d $SPLUNKHOME/var/lib/splunk/fishbucket/splunkprivate_db/ -k ALL | wc -l

0 Karma
Highlighted

Re: How much splunk memory use should I expect while monitoring a directory with many entries?

Path Finder

I had to modify your command since I'm on Windows: %SPLUNKHOME%\bin\splunk.exe cmd btprobe -d %SPLUNKHOME%\var\lib\splunk\fishbucket\splunkprivatedb\ -k ALL | find /c "key". The result is 66827.

0 Karma
Highlighted

Re: How much splunk memory use should I expect while monitoring a directory with many entries?

Splunk Employee
Splunk Employee

We believe that Splunk monitor uses on the order of 1KB per file or directory in a monitored location. This includes files that will not be read.

Highlighted

Re: How much splunk memory use should I expect while monitoring a directory with many entries?

Communicator

Try to clean out your fishbucket.

%SPLUNK_HOME%/bin/splunk stop 
%SPLUNK_HOME%/bin/splunk clean eventdata -index _fishbucket 
%SPLUNK_HOME%/bin/splunk start
0 Karma