How much splunk memory use should I expect while monitoring a directory with many entries?

[andyk]

If I remove the only input stanza I have on a forwarder and restart Splunk the memory usage is 2GB. How can a forwarder use that much memory if I don't have any input stanzas?

On this page it's described how Splunk use CRC checksums to keep track of which files it has seen before.

http://docs.splunk.com/Documentation/Splunk/latest/Data/Howlogfilerotationishandled

How much space does this data use on disk and in memory if you have let Splunk monitor a directory that over the years have contained more than 1 000 000 files?

This question is related to this question:
http://splunk-base.splunk.com/answers/32528/lots-of-log-files-how-can-i-reduce-forwarder-memory-usag...

[peter_krammer]

Try to clean out your fishbucket.

%SPLUNK_HOME%/bin/splunk stop 
%SPLUNK_HOME%/bin/splunk clean eventdata -index _fishbucket 
%SPLUNK_HOME%/bin/splunk start

[jrodman]

We believe that Splunk monitor uses on the order of 1KB per file or directory in a monitored location. This includes files that will not be read.

[andyk]

I had to modify your command since I'm on Windows: %SPLUNK_HOME%\bin\splunk.exe cmd btprobe -d %SPLUNK_HOME%\var\lib\splunk\fishbucket\splunk_private_db\ -k ALL | find /c "key". The result is 66827.

[hexx]

Hi andyw. A few questions for you :

- How do you measure the memory consumption of the forwarder? Which metric (Vsize? RSS?) shows the 2Gb figure?

- How large is the directory $SPLUNK_HOME/var/lib/splunk/fishbucket on the forwarder?
- How many records exist in your fishbucket? To find out, run :

$SPLUNK_HOME/bin/splunk cmd btprobe -d $SPLUNK_HOME/var/lib/splunk/fishbucket/splunk_private_db/ -k ALL | wc -l