How to Show Indexer Activity

[brosselle]

I'm wondering if there is a command line way to show search activity on the indexers. For example, I had a situation where there were no running jobs on the search head, yet my indexers were obviously running a search/task that was hammering the disks.

Is there a way to show that activity? I was thinking I could just connect to the web portal on the indexer and show running jobs, but I was hoping there was a command line way to it.

[javiergn]

If you are running 6.2 or older you can probably find what you are looking for from the DMC:

Settings > Distributed Management Console > Indexing > Performance > Indexing Performance: Instance

[brosselle]

That does provide a lot of good info. I don't think it has what I'm looking for though. Here's what happened in my case:

User writes a very resource intensive query and launches it. When it they see it's going to take a REALLY long time, they change the query slightly and then launch that. Repeat that process a dozen times.

When I find out things are going sideways, I log into the search head and see the queries running and realize what's going on.

Here's the weird part...
Even when they all show complete (or killed) under activity, I still have massive disk i/o on the indexers that lasts for awhile longer.

So, what I really want to see is what's happening on the indexer during this time. The DMC will show me that it's getting hammered, but I don't think it will show me the hammer, and a way to potentially kill it, if that's even possible. I ended up restarting Splunk on the indexer to kill it.

[javiergn]

See if this helps in that case:

http://docs.splunk.com/Documentation/Splunk/6.4.2/Search/ManagejobsfromtheOS