How many Splunk processes are normal on a Linux indexer?

[daniel_hanft]

Hi Splunk Community,

how many splunk processes are normal on a Linux Indexer? I've observed sometimes there are up to 37 processes on one system (when using the command: # ps uax | grep splunk).
Can someone tell me a good threshold value we can configure in our system monitoring tool for alerting?

Many thanks in advance.

Daniel

[sowings]

Typically there's one monolithic splunkd process, then two for each running search (a helper and the actual searcher). These may show "rt" in the search name if they are real time searches. Additionally, as @jeremiahc4 points out, other maintenance processes may start up additional copies of splunkd.

[jeremiahc4 · Accepted solution]

That number depends on how many searches you may be running. I seem to only have 2 splunkd processes running which aren't specific searches. Occasionally I'll see another pop up when rolling buckets for instance. Try this command to narrow the field unless you are interested in how many searches are in process.

 ps uax | grep splunkd | grep -v grep | grep -v search

[daniel_hanft]

Thank you jeremiahc4. When I type your command, I get a total number of 3 processes running.

The output is this:

splunk    9338 21.0  0.0 948748 79344 ?        Sl   Oct17 1845:41 splunkd -p 8089 restart

splunk    9339  0.0  0.0  49236  3428 ?        Ss   Oct17   5:50 [splunkd pid=9338] splunkd -p 8089 restart [process-runner]

splunk    9406  0.0  0.0  49192 11692 ?        Ss   Oct17   7:10 /opt/splunk/bin/splunkd instrument-resource-usage

So can I assume a number of 3 processes is normal on an Splunk Indexer?

[jeremiahc4]

The first two are constant (splunkd -p 8089...). The third looks like a maintenance process and might not be there all the time. I'd go with 2 minimum for your process monitor (i.e. greater than 2 = good).

[daniel_hanft]

Thank you @jeremiahc4 🙂