Getting Data In

How much splunk memory use should I expect while monitoring a directory with many entries?

andyk
Path Finder

If I remove the only input stanza I have on a forwarder and restart Splunk the memory usage is 2GB. How can a forwarder use that much memory if I don't have any input stanzas?

On this page it's described how Splunk use CRC checksums to keep track of which files it has seen before.

http://docs.splunk.com/Documentation/Splunk/latest/Data/Howlogfilerotationishandled

How much space does this data use on disk and in memory if you have let Splunk monitor a directory that over the years have contained more than 1 000 000 files?

This question is related to this question:
http://splunk-base.splunk.com/answers/32528/lots-of-log-files-how-can-i-reduce-forwarder-memory-usag...

Tags (1)

peter_krammer
Communicator

Try to clean out your fishbucket.

%SPLUNK_HOME%/bin/splunk stop 
%SPLUNK_HOME%/bin/splunk clean eventdata -index _fishbucket 
%SPLUNK_HOME%/bin/splunk start
0 Karma

jrodman
Splunk Employee
Splunk Employee

We believe that Splunk monitor uses on the order of 1KB per file or directory in a monitored location. This includes files that will not be read.

andyk
Path Finder

I had to modify your command since I'm on Windows: %SPLUNK_HOME%\bin\splunk.exe cmd btprobe -d %SPLUNK_HOME%\var\lib\splunk\fishbucket\splunk_private_db\ -k ALL | find /c "key". The result is 66827.

0 Karma

hexx
Splunk Employee
Splunk Employee

Hi andyw. A few questions for you :

- How do you measure the memory consumption of the forwarder? Which metric (Vsize? RSS?) shows the 2Gb figure?

- How large is the directory $SPLUNK_HOME/var/lib/splunk/fishbucket on the forwarder?
- How many records exist in your fishbucket? To find out, run :

$SPLUNK_HOME/bin/splunk cmd btprobe -d $SPLUNK_HOME/var/lib/splunk/fishbucket/splunk_private_db/ -k ALL | wc -l

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to November Tech Talks, Office Hours, and Webinars!

🍂 Fall into November with a fresh lineup of Community Office Hours, Tech Talks, and Webinars we’ve ...

Transform your security operations with Splunk Enterprise Security

Hi Splunk Community, Splunk Platform has set a great foundation for your security operations. With the ...

Splunk Admins and App Developers | Earn a $35 gift card!

Splunk, in collaboration with ESG (Enterprise Strategy Group) by TechTarget, is excited to announce a ...