Splunk indexer server run out of memory

[arun_kant_sharm]

Hi Experts,

My splunk indexer server are running out if memory, its main reason are 

/opt/splunk/var/run/searchpeers
/opt/splunk/var/lib/splunk/_introspection
/opt/splunk/var/lib/splunk/_internaldb
/opt/splunk/var/lib/splunk/kvstore 

Indexer _introspection, _internaldb,  kvstore have default setting, its data are not move in cold and frozen bucket.

Please suggest what can I do to create space at my server ?

[richgalloway]

Your problem is disk space, not memory.  Well, you may also have a memory problem, but the evidence in your posting screams disk space.

A related problem is Splunk and the operating system are sharing a file system (/).  This can become an issue when Splunk uses up disk space and prevents the OS from doing any work.  This is not called out in Splunk docs, but is a basic Linux admin matter.  /, $SPLUNK_HOME, and $SPLUNK_DB should be separate file systems.

Don't touch anything in /opt/splunk/var/lib/splunk.  That's where your data is stored and you risk data loss by manipulating files there.

You can safely delete files in /opt/splunk/var/log/splunk having names that end with a digit.

You really have two options: 1) add more storage; or 2) use the frozenTimePeriodInSeconds setting in indexes.conf to reduce the retention time of your indexed data.  I recommend the former.

[koshyk]

Your question is big vague

- Are you running out of memory or Disk space? the screenshot seems to show problem with disk-space

- Indexer needs quite lot of memory for powerful systems or large data. What's your spec for memory? How much RAM?

- What's your indexes.conf specifications (run a btool and put the output in your Question, so people could understand the issues)