topic Re: What is Splunkd Doing in Monitoring Splunk

What is Splunkd Doing

hartfoml — Mon, 28 Sep 2020 16:24:55 GMT

My indexer is running at 97,8% CPU utalisation and there are about 18 splunkd threads running along with python and splunk_optimizer and ssl_test.pl

How can I tell what all the Splunkd threads are doing and if I can kill some of them to ease the load or at least look at what they are so that i can schedule them to run at different times.

Thanks for your help... 🙂

Re: What is Splunkd Doing

linu1988 — Thu, 17 Apr 2014 15:37:06 GMT

Re: What is Splunkd Doing

hartfoml — Thu, 17 Apr 2014 15:45:24 GMT

I logged into the indexer https://MyIndexer:8000 as admin opend the jobs window selected all apps All Owners Status Running ans even though Top shows several Splunkd processes running no jods or at time only a few jobs are running.

thanks this helps a little to see what scheduled or search jobs are running on the indexer

Re: What is Splunkd Doing

linu1988 — Thu, 17 Apr 2014 15:50:00 GMT

its not the indexer, the search head would initiate the searches.. other indexing activity would also be going on..

Re: What is Splunkd Doing

hartfoml — Thu, 17 Apr 2014 16:22:24 GMT

Thanks this helps a bit. I was hoping for a way to know which pid was running which job. Something with btools or REST API or something that would tell me which splunkd process is running which job.

Thanks for the help I really appreciate it.

Re: What is Splunkd Doing

MuS — Fri, 18 Apr 2014 06:13:38 GMT

run $SPLUNK_HOME/bin/splunk status to get back a pid list of main splunkd process, splunkweb process and the splunkd helper processes (your searches)

Re: What is Splunkd Doing

yannK — Tue, 22 Apr 2014 01:53:06 GMT

To check if they are search processes, install the SOS app, and enable the ps_sos scripts (for linux or windows with powershell)
then check the resource usage dashboard of the APP for the details.

To check the scheduler, use the SOS app too, there is a scheduled searches dashboard.

Re: What is Splunkd Doing

hartfoml — Thu, 24 Apr 2014 15:20:20 GMT

thanks Yann,

This was very good advice. Yesterday I had a busy day and the virtual memory went way high and then dropped down after the work was done. The SOS app does not map PID numbers to searches so I can not see witch Splunkd process is tied to which user/search.

Also the Swap file on my indexer was never released and I had to bounce splunk to get the swap file freed up. I find that when the swap file is close to 100% utilized the server is likely to crash. RHEL 5.x

Re: What is Splunkd Doing

yannK — Thu, 24 Apr 2014 17:36:34 GMT

red-hat and swap, make me think of this known issue, http://docs.splunk.com/Documentation/Splunk/6.0.3/ReleaseNotes/SplunkandTHP