Getting Data In

How do I find what is causing my typing queue blockage?

Splunk Employee
Splunk Employee

How do I find sources/source types/hosts/indexes causing typing queue blockage?

Tags (2)
1 Solution

Splunk Employee
Splunk Employee

Steps
1) Set under [default] stanza in limits.conf

regexcpuprofiling = true

regexcpuprofiling =
* Enable CPU time metrics for RegexProcessor. Output will be in the
metrics.log file.
Entries in metrics.log will appear perhostregexcpu, persourceregexcpu,
persourcetyperegexcpu, perindexregexcpu.
* Default: false

2) Set under [metrics] stanza in limits.conf

maxseries = 50

maxseries =
* The number of series to include in the perxthruput reports in metrics.log.
* Default: 10

3) restart splunk

4) Wait for typing queue to block.

5) Goto splunk UI and following queries will be helpful:

Which source type is taking most of the cpu time.

index=_internal host= source=*metrics.log group=per_sourcetype_regex_cpu |timechart max(cpu) by series

Which source type is taking most of the cpu time per event:

index=_internal host= source=*metrics.log group=per_sourcetype_regex_cpu |timechart max(cpupe) by series

Repeat queries for perhostregexcpu, persourceregexcpu, and perindexregex_cpu(if needed)

cpu > total cpu time for a given series
cpupe > cpu time per event for a given series
bytes > total bytes processes for a given series
ev > total events for a given series

View solution in original post

Splunk Employee
Splunk Employee

Steps
1) Set under [default] stanza in limits.conf

regexcpuprofiling = true

regexcpuprofiling =
* Enable CPU time metrics for RegexProcessor. Output will be in the
metrics.log file.
Entries in metrics.log will appear perhostregexcpu, persourceregexcpu,
persourcetyperegexcpu, perindexregexcpu.
* Default: false

2) Set under [metrics] stanza in limits.conf

maxseries = 50

maxseries =
* The number of series to include in the perxthruput reports in metrics.log.
* Default: 10

3) restart splunk

4) Wait for typing queue to block.

5) Goto splunk UI and following queries will be helpful:

Which source type is taking most of the cpu time.

index=_internal host= source=*metrics.log group=per_sourcetype_regex_cpu |timechart max(cpu) by series

Which source type is taking most of the cpu time per event:

index=_internal host= source=*metrics.log group=per_sourcetype_regex_cpu |timechart max(cpupe) by series

Repeat queries for perhostregexcpu, persourceregexcpu, and perindexregex_cpu(if needed)

cpu > total cpu time for a given series
cpupe > cpu time per event for a given series
bytes > total bytes processes for a given series
ev > total events for a given series

View solution in original post

SplunkTrust
SplunkTrust

This is a fantastic post. The only thing I would add is that the regex_cpu_profiling was added in 6.6. Thanks!

0 Karma

Splunk Employee
Splunk Employee

It's integrated with DMC as well starting 7.x. However enabling regexcpuprofiling is required.

SplunkTrust
SplunkTrust

Didn't know that. This post is solid gold!

0 Karma