Troubleshooting Splunk Queues (Typing Queue)

[mbrunetto]

My Typing Queue is currently blocking and causing backups. I believe I have the order right
udpin/splunktcpin, parsing, and agg queues are all backed up. Indexing queue has some localized spikes, but is mostly at 0. This should indicate a delay in the Typing Pipeline. My data comes in waves with the workday, and the queues max during the workday, and clear out overnight.

Where would I go next to try and clear these queues out? What are my troubleshooting steps? It looks like this pipeline is trying to do regex's and punctuation; but how do I see what part of the pipeline is holding up the queue? I'd like to find out if it's something that I've put in, and if so, which thing to remove.

Since the index seems unblocked, I don't think this has anything to do with my disk speed. My CPUs (8) are busy, but not overworked, and I have plenty of free memory. I run a single box doing indexer/search on 10G of data/day.

[phoffman_splunk]

1st easiest thing to start with is to download and install the S.o.S app (app link here) If you install this on your search head, remember to deploy the TA (Links here on the documentation tab) to your indexer(s).

In the S.o.S. app, check out the "Estimated percentage of total CPU used per Splunk processor" panel under the "Indexing Performance" dashboard. This will let you view where most of your CPU processing time is going. most typically it is a bad regex.

Then it is a matter of finding the bad regex that was put in place, through exploring your transorms.conf settings through the S.o.S. "Configuration File Viewer" view.