topic Re: Splunk indexer server run out of memory in Getting Data In

Splunk indexer server run out of memory

arun_kant_sharm — Thu, 11 Jun 2020 11:45:02 GMT

Hi Experts,

My splunk indexer server are running out if memory, its main reason are

/opt/splunk/var/run/searchpeers
/opt/splunk/var/lib/splunk/_introspection
/opt/splunk/var/lib/splunk/_internaldb
/opt/splunk/var/lib/splunk/kvstore

Indexer _introspection, _internaldb, kvstore have default setting, its data are not move in cold and frozen bucket.

Please suggest what can I do to create space at my server ?

Re: Splunk indexer server run out of memory

koshyk — Fri, 12 Jun 2020 12:44:07 GMT

Your question is big vague

- Are you running out of memory or Disk space? the screenshot seems to show problem with disk-space

- Indexer needs quite lot of memory for powerful systems or large data. What's your spec for memory? How much RAM?

- What's your indexes.conf specifications (run a btool and put the output in your Question, so people could understand the issues)

Re: Splunk indexer server run out of memory

richgalloway — Fri, 12 Jun 2020 13:12:36 GMT

Your problem is disk space, not memory. Well, you may also have a memory problem, but the evidence in your posting screams disk space.

A related problem is Splunk and the operating system are sharing a file system (/). This can become an issue when Splunk uses up disk space and prevents the OS from doing any work. This is not called out in Splunk docs, but is a basic Linux admin matter. /, $SPLUNK_HOME, and $SPLUNK_DB should be separate file systems.

Don't touch anything in /opt/splunk/var/lib/splunk. That's where your data is stored and you risk data loss by manipulating files there.

You can safely delete files in /opt/splunk/var/log/splunk having names that end with a digit.

You really have two options: 1) add more storage; or 2) use the frozenTimePeriodInSeconds setting in indexes.conf to reduce the retention time of your indexed data. I recommend the former.