https://community.splunk.com/t5/Getting-Data-In/How-much-splunk-memory-use-should-I-expect-while-monitoring-a/m-p/98194#M20496 <P>We believe that Splunk monitor uses on the order of 1KB per file or directory in a monitored location. This includes files that will not be read.</P> Fri, 10 Oct 2014 09:24:02 GMT jrodman 2014-10-10T09:24:02Z https://community.splunk.com/t5/Getting-Data-In/How-much-splunk-memory-use-should-I-expect-while-monitoring-a/m-p/98191#M20493 <P>If I remove the only input stanza I have on a forwarder and restart Splunk the memory usage is 2GB. How can a forwarder use that much memory if I don't have any input stanzas?</P> <P>On this page it's described how Splunk use CRC checksums to keep track of which files it has seen before. </P> <P><A href="http://docs.splunk.com/Documentation/Splunk/latest/Data/Howlogfilerotationishandled">http://docs.splunk.com/Documentation/Splunk/latest/Data/Howlogfilerotationishandled</A></P> <P>How much space does this data use on disk and in memory if you have let Splunk monitor a directory that over the years have contained more than 1 000 000 files? </P> <P>This question is related to this question:<BR /> <A href="http://splunk-base.splunk.com/answers/32528/lots-of-log-files-how-can-i-reduce-forwarder-memory-usage">http://splunk-base.splunk.com/answers/32528/lots-of-log-files-how-can-i-reduce-forwarder-memory-usage</A></P> Wed, 26 Oct 2011 12:54:40 GMT https://community.splunk.com/t5/Getting-Data-In/How-much-splunk-memory-use-should-I-expect-while-monitoring-a/m-p/98191#M20493 andyk 2011-10-26T12:54:40Z https://community.splunk.com/t5/Getting-Data-In/How-much-splunk-memory-use-should-I-expect-while-monitoring-a/m-p/98192#M20494 <P>Hi andyw. A few questions for you :<BR /><BR /> - How do you measure the memory consumption of the forwarder? Which metric (Vsize? RSS?) shows the 2Gb figure?<BR /><BR /> - How large is the directory $SPLUNK_HOME/var/lib/splunk/fishbucket on the forwarder?<BR /> - How many records exist in your fishbucket? To find out, run :<BR /><BR /> $SPLUNK_HOME/bin/splunk cmd btprobe -d $SPLUNK_HOME/var/lib/splunk/fishbucket/splunk_private_db/ -k ALL | wc -l</P> Mon, 28 Sep 2020 10:01:40 GMT https://community.splunk.com/t5/Getting-Data-In/How-much-splunk-memory-use-should-I-expect-while-monitoring-a/m-p/98192#M20494 hexx 2020-09-28T10:01:40Z https://community.splunk.com/t5/Getting-Data-In/How-much-splunk-memory-use-should-I-expect-while-monitoring-a/m-p/98193#M20495 <P>I had to modify your command since I'm on Windows: %SPLUNK_HOME%\bin\splunk.exe cmd btprobe -d %SPLUNK_HOME%\var\lib\splunk\fishbucket\splunk_private_db\ -k ALL | find /c "key". The result is 66827.</P> Mon, 28 Sep 2020 10:01:57 GMT https://community.splunk.com/t5/Getting-Data-In/How-much-splunk-memory-use-should-I-expect-while-monitoring-a/m-p/98193#M20495 andyk 2020-09-28T10:01:57Z https://community.splunk.com/t5/Getting-Data-In/How-much-splunk-memory-use-should-I-expect-while-monitoring-a/m-p/98194#M20496 <P>We believe that Splunk monitor uses on the order of 1KB per file or directory in a monitored location. This includes files that will not be read.</P> Fri, 10 Oct 2014 09:24:02 GMT https://community.splunk.com/t5/Getting-Data-In/How-much-splunk-memory-use-should-I-expect-while-monitoring-a/m-p/98194#M20496 jrodman 2014-10-10T09:24:02Z https://community.splunk.com/t5/Getting-Data-In/How-much-splunk-memory-use-should-I-expect-while-monitoring-a/m-p/98195#M20497 <P>Try to clean out your fishbucket. </P> <PRE><CODE>%SPLUNK_HOME%/bin/splunk stop %SPLUNK_HOME%/bin/splunk clean eventdata -index _fishbucket %SPLUNK_HOME%/bin/splunk start </CODE></PRE> Fri, 10 Oct 2014 09:28:07 GMT https://community.splunk.com/t5/Getting-Data-In/How-much-splunk-memory-use-should-I-expect-while-monitoring-a/m-p/98195#M20497 peter_krammer 2014-10-10T09:28:07Z