Splunk Search

How many Splunk processes are normal on a Linux indexer?

daniel_hanft
Explorer

Hi Splunk Community,

how many splunk processes are normal on a Linux Indexer? I've observed sometimes there are up to 37 processes on one system (when using the command: # ps uax | grep splunk).
Can someone tell me a good threshold value we can configure in our system monitoring tool for alerting?

Many thanks in advance.

Daniel

1 Solution

jeremiahc4
Builder

That number depends on how many searches you may be running. I seem to only have 2 splunkd processes running which aren't specific searches. Occasionally I'll see another pop up when rolling buckets for instance. Try this command to narrow the field unless you are interested in how many searches are in process.

 ps uax | grep splunkd | grep -v grep | grep -v search

View solution in original post

sowings
Splunk Employee
Splunk Employee

Typically there's one monolithic splunkd process, then two for each running search (a helper and the actual searcher). These may show "rt" in the search name if they are real time searches. Additionally, as @jeremiahc4 points out, other maintenance processes may start up additional copies of splunkd.

jeremiahc4
Builder

That number depends on how many searches you may be running. I seem to only have 2 splunkd processes running which aren't specific searches. Occasionally I'll see another pop up when rolling buckets for instance. Try this command to narrow the field unless you are interested in how many searches are in process.

 ps uax | grep splunkd | grep -v grep | grep -v search

View solution in original post

daniel_hanft
Explorer

Thank you jeremiahc4. When I type your command, I get a total number of 3 processes running.

The output is this:

splunk    9338 21.0  0.0 948748 79344 ?        Sl   Oct17 1845:41 splunkd -p 8089 restart

splunk    9339  0.0  0.0  49236  3428 ?        Ss   Oct17   5:50 [splunkd pid=9338] splunkd -p 8089 restart [process-runner]

splunk    9406  0.0  0.0  49192 11692 ?        Ss   Oct17   7:10 /opt/splunk/bin/splunkd instrument-resource-usage

So can I assume a number of 3 processes is normal on an Splunk Indexer?

0 Karma

jeremiahc4
Builder

The first two are constant (splunkd -p 8089...). The third looks like a maintenance process and might not be there all the time. I'd go with 2 minimum for your process monitor (i.e. greater than 2 = good).

0 Karma

daniel_hanft
Explorer

Thank you @jeremiahc4 🙂

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.