Hi Splunk Community,
how many splunk processes are normal on a Linux Indexer? I've observed sometimes there are up to 37 processes on one system (when using the command: # ps uax | grep splunk).
Can someone tell me a good threshold value we can configure in our system monitoring tool for alerting?
Many thanks in advance.
Daniel
That number depends on how many searches you may be running. I seem to only have 2 splunkd processes running which aren't specific searches. Occasionally I'll see another pop up when rolling buckets for instance. Try this command to narrow the field unless you are interested in how many searches are in process.
ps uax | grep splunkd | grep -v grep | grep -v search
Typically there's one monolithic splunkd process, then two for each running search (a helper and the actual searcher). These may show "rt" in the search name if they are real time searches. Additionally, as @jeremiahc4 points out, other maintenance processes may start up additional copies of splunkd.
That number depends on how many searches you may be running. I seem to only have 2 splunkd processes running which aren't specific searches. Occasionally I'll see another pop up when rolling buckets for instance. Try this command to narrow the field unless you are interested in how many searches are in process.
ps uax | grep splunkd | grep -v grep | grep -v search
Thank you jeremiahc4. When I type your command, I get a total number of 3 processes running.
The output is this:
splunk 9338 21.0 0.0 948748 79344 ? Sl Oct17 1845:41 splunkd -p 8089 restart
splunk 9339 0.0 0.0 49236 3428 ? Ss Oct17 5:50 [splunkd pid=9338] splunkd -p 8089 restart [process-runner]
splunk 9406 0.0 0.0 49192 11692 ? Ss Oct17 7:10 /opt/splunk/bin/splunkd instrument-resource-usage
So can I assume a number of 3 processes is normal on an Splunk Indexer?
The first two are constant (splunkd -p 8089...). The third looks like a maintenance process and might not be there all the time. I'd go with 2 minimum for your process monitor (i.e. greater than 2 = good).
Thank you @jeremiahc4 🙂