Monitoring Splunk

deployment monitor MB received SLOOOOOOWWWW

mcbradford
Contributor

If I run the All Sourcetypes dashboard, the MB received panel for the past 24 hours, the panel takes just over nine minutes to complete. I studied the search and it is made up of three macros that form this search:

index="_internal" source="*license_usage.lo*" type!=*Summary | eval lastReceived = _time | rename s as source st as mysourcetype h as host b as bytes o as originator | eval my_splunk_server = splunk_server | fields lastReceived source mysourcetype host bytes pool originator my_splunk_server source | bin _time span=10m | stats sum(bytes) as bytes max(lastReceived) as lastReceived by mysourcetype _time pool host | eval kb = bytes/1024 | eval mb = kb/1024 |timechart minspan=10m bins=200 sum(mb) as mbytes by mysourcetype

If I run this search manually the results are returned within 1 minute.

Any idea about what is going on????

0 Karma

mkinsley_splunk
Splunk Employee
Splunk Employee

The macro powering the search in the "MB Recevied" Panel is:

sourcetype_metrics_timechart

You can see what is consuming all the time in your search by inspecting the job. Here is what you'll want to do:

  1. Reload the "All Sourcetypes" panel and go on a small coffee break ( not too long of a break or the job details will get cleaned up).

  2. Click on the "Jobs" link in the upper right corner

  3. Click "Inspect" on the entry for "sourcetype_metrics_timechart"

It sounds like the search might not be using Report Acceleration correctly. Do you see a message indicating that sumaries are being used?

It would look something like the following:

DEBUG: [my.host.name] Using summaries for search, 

If Search summaries are being used, then you may have run into a bug in the core product with search acceleration. In that case, I would recommend opening a support case.

araitz
Splunk Employee
Splunk Employee

Not enough information. Can you open a support case please?

0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...