Getting Data In

Troubleshooting Splunk Queues (Typing Queue)

mbrunetto
Path Finder

My Typing Queue is currently blocking and causing backups. I believe I have the order right
udpin/splunktcpin, parsing, and agg queues are all backed up. Indexing queue has some localized spikes, but is mostly at 0. This should indicate a delay in the Typing Pipeline. My data comes in waves with the workday, and the queues max during the workday, and clear out overnight.

Where would I go next to try and clear these queues out? What are my troubleshooting steps? It looks like this pipeline is trying to do regex's and punctuation; but how do I see what part of the pipeline is holding up the queue? I'd like to find out if it's something that I've put in, and if so, which thing to remove.

Since the index seems unblocked, I don't think this has anything to do with my disk speed. My CPUs (8) are busy, but not overworked, and I have plenty of free memory. I run a single box doing indexer/search on 10G of data/day.

phoffman_splunk
Splunk Employee
Splunk Employee

1st easiest thing to start with is to download and install the S.o.S app (app link here) If you install this on your search head, remember to deploy the TA (Links here on the documentation tab) to your indexer(s).

In the S.o.S. app, check out the "Estimated percentage of total CPU used per Splunk processor" panel under the "Indexing Performance" dashboard. This will let you view where most of your CPU processing time is going. most typically it is a bad regex.

Then it is a matter of finding the bad regex that was put in place, through exploring your transorms.conf settings through the S.o.S. "Configuration File Viewer" view.

Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...