Monitoring Splunk

Any idea what is causing high memory usage in indexers?

Ashwini008
Builder

We have distributed environment with 4 Splunk Indexers which are consuming high memory . It reaches to 100% and remains unreachable until we restart splunkd service. Once restarted, memory comes down and the same process repeats on other indexers within a span of couple of hours.

64GB of Physical Memory is available on each indexer and saved searches/Scheduled searches are not consuming high memory. Unable to understand why there is spike in the memory usage. 

In DMC It shows,Splunkd server is using high Physical Memory usage by process class. PID keeps increasing as below. Please suggest how do i find the root cause for this issue and how to fix it

Ashwini008_1-1670496771812.png

 



Ashwini008_0-1670496350966.png

 

Labels (3)
Tags (3)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @Ashwini008,

maybe, but anyway, open a Case to Splunk Support, they can find and hint how to solve the issue.

Ciao.

Giuseppe

View solution in original post

ssanplunk
Path Finder

Hi.

If your splunk version is 9.1 or higher, please refer to the case below.
You can solve it by setting the option below in server.conf to false.
> https://splunk.my.site.com/customer/s/article/PreforkedSearchProcessException-can-t-launch-new-searc...

However, since the default setting is true, it is recommended to contact splunk support and decide.

[general]
enable_search_process_long_lifespan = false

0 Karma

ssanplunk
Path Finder

Did you solve this problem?

I had the same symptoms and was wondering how you solved it.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Ashwini008,

there's no apparent reason because 64 GB RAM aren't sufficient for an Indexer.

open a Case to Splunk Support.

Ciao.

Giuseppe

0 Karma

Ashwini008
Builder

Hi @gcusello ,

 

We haven't faced this issue from past 4 years.We have been using the same specifications till now.

There is no increase in the indexing data. We recently copied splunk_home/var/lib/splunk data (this was backed up data) into 4 indexers , I also see that  there are excess buckets. Could this be an reason? 

Ashwini008_0-1670497277417.png

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Ashwini008,

maybe, but anyway, open a Case to Splunk Support, they can find and hint how to solve the issue.

Ciao.

Giuseppe

gcusello
SplunkTrust
SplunkTrust

Hi @Ashwini008,

if one answer solves your need, please accept one answer for the other people of Community or tell us how we can help you.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

0 Karma
Get Updates on the Splunk Community!

Transforming Financial Data into Fraud Intelligence

Every day, banks and financial companies handle millions of transactions, logins, and customer interactions ...

What's New in Splunk Observability Cloud and Splunk AppDynamics - May 2025

This month, we’re delivering several new innovations in Splunk Observability Cloud and Splunk AppDynamics ...

How to send events & findings from AWS to Splunk using Amazon EventBridge

Amazon EventBridge is a serverless service that uses events to connect application components together, making ...