Getting Data In

Splunk indexer server run out of memory

arun_kant_sharm
Path Finder

Hi Experts,

My splunk indexer server are running out if memory, its main reason are 

/opt/splunk/var/run/searchpeers
/opt/splunk/var/lib/splunk/_introspection
/opt/splunk/var/lib/splunk/_internaldb
/opt/splunk/var/lib/splunk/kvstore 

arun_kant_sharm_1-1591875437534.png

Indexer _introspection, _internaldb,  kvstore have default setting, its data are not move in cold and frozen bucket.

Please suggest what can I do to create space at my server ?

 

 

Labels (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Your problem is disk space, not memory.  Well, you may also have a memory problem, but the evidence in your posting screams disk space.

A related problem is Splunk and the operating system are sharing a file system (/).  This can become an issue when Splunk uses up disk space and prevents the OS from doing any work.  This is not called out in Splunk docs, but is a basic Linux admin matter.  /, $SPLUNK_HOME, and $SPLUNK_DB should be separate file systems.

Don't touch anything in /opt/splunk/var/lib/splunk.  That's where your data is stored and you risk data loss by manipulating files there.

You can safely delete files in /opt/splunk/var/log/splunk having names that end with a digit.

You really have two options: 1) add more storage; or 2) use the frozenTimePeriodInSeconds setting in indexes.conf to reduce the retention time of your indexed data.  I recommend the former.

---
If this reply helps you, Karma would be appreciated.

koshyk
Super Champion

Your question is big vague

- Are you running out of memory or Disk space? the screenshot seems to show problem with disk-space

- Indexer needs quite lot of memory for powerful systems or large data. What's your spec for memory? How much RAM?

- What's your indexes.conf specifications (run a btool and put the output in your Question, so people could understand the issues)

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...