topic Re: How many Splunk processes are normal on a Linux indexer? in Splunk Search

How many Splunk processes are normal on a Linux indexer?

daniel_hanft — Wed, 22 Oct 2014 12:32:28 GMT

Hi Splunk Community,

how many splunk processes are normal on a Linux Indexer? I've observed sometimes there are up to 37 processes on one system (when using the command: # ps uax | grep splunk).
Can someone tell me a good threshold value we can configure in our system monitoring tool for alerting?

Many thanks in advance.

Daniel

Re: How many Splunk processes are normal on a Linux indexer?

jeremiahc4 — Wed, 22 Oct 2014 15:01:49 GMT

That number depends on how many searches you may be running. I seem to only have 2 splunkd processes running which aren't specific searches. Occasionally I'll see another pop up when rolling buckets for instance. Try this command to narrow the field unless you are interested in how many searches are in process.

 ps uax | grep splunkd | grep -v grep | grep -v search

Re: How many Splunk processes are normal on a Linux indexer?

sowings — Wed, 22 Oct 2014 16:29:47 GMT

Typically there's one monolithic splunkd process, then two for each running search (a helper and the actual searcher). These may show "rt" in the search name if they are real time searches. Additionally, as @jeremiahc4 points out, other maintenance processes may start up additional copies of splunkd.

Re: How many Splunk processes are normal on a Linux indexer?

daniel_hanft — Thu, 23 Oct 2014 12:21:24 GMT

Thank you jeremiahc4. When I type your command, I get a total number of 3 processes running.

The output is this:

splunk    9338 21.0  0.0 948748 79344 ?        Sl   Oct17 1845:41 splunkd -p 8089 restart

splunk    9339  0.0  0.0  49236  3428 ?        Ss   Oct17   5:50 [splunkd pid=9338] splunkd -p 8089 restart [process-runner]

splunk    9406  0.0  0.0  49192 11692 ?        Ss   Oct17   7:10 /opt/splunk/bin/splunkd instrument-resource-usage

So can I assume a number of 3 processes is normal on an Splunk Indexer?

Re: How many Splunk processes are normal on a Linux indexer?

jeremiahc4 — Thu, 23 Oct 2014 18:07:50 GMT

The first two are constant (splunkd -p 8089...). The third looks like a maintenance process and might not be there all the time. I'd go with 2 minimum for your process monitor (i.e. greater than 2 = good).

Re: How many Splunk processes are normal on a Linux indexer?

daniel_hanft — Fri, 24 Oct 2014 06:31:57 GMT

Thank you @jeremiahc4 🙂