Monitoring Splunk

How to Show Indexer Activity

brosselle
New Member

I'm wondering if there is a command line way to show search activity on the indexers. For example, I had a situation where there were no running jobs on the search head, yet my indexers were obviously running a search/task that was hammering the disks.

Is there a way to show that activity? I was thinking I could just connect to the web portal on the indexer and show running jobs, but I was hoping there was a command line way to it.

Tags (1)
0 Karma

javiergn
SplunkTrust
SplunkTrust

If you are running 6.2 or older you can probably find what you are looking for from the DMC:

Settings > Distributed Management Console > Indexing > Performance > Indexing Performance: Instance

0 Karma

brosselle
New Member

That does provide a lot of good info. I don't think it has what I'm looking for though. Here's what happened in my case:

User writes a very resource intensive query and launches it. When it they see it's going to take a REALLY long time, they change the query slightly and then launch that. Repeat that process a dozen times.

When I find out things are going sideways, I log into the search head and see the queries running and realize what's going on.

Here's the weird part...
Even when they all show complete (or killed) under activity, I still have massive disk i/o on the indexers that lasts for awhile longer.

So, what I really want to see is what's happening on the indexer during this time. The DMC will show me that it's getting hammered, but I don't think it will show me the hammer, and a way to potentially kill it, if that's even possible. I ended up restarting Splunk on the indexer to kill it.

0 Karma

javiergn
SplunkTrust
SplunkTrust
0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!