Monitoring Splunk

deployment monitor MB received SLOOOOOOWWWW

mcbradford
Contributor

If I run the All Sourcetypes dashboard, the MB received panel for the past 24 hours, the panel takes just over nine minutes to complete. I studied the search and it is made up of three macros that form this search:

index="_internal" source="*license_usage.lo*" type!=*Summary | eval lastReceived = _time | rename s as source st as mysourcetype h as host b as bytes o as originator | eval my_splunk_server = splunk_server | fields lastReceived source mysourcetype host bytes pool originator my_splunk_server source | bin _time span=10m | stats sum(bytes) as bytes max(lastReceived) as lastReceived by mysourcetype _time pool host | eval kb = bytes/1024 | eval mb = kb/1024 |timechart minspan=10m bins=200 sum(mb) as mbytes by mysourcetype

If I run this search manually the results are returned within 1 minute.

Any idea about what is going on????

0 Karma

mkinsley_splunk
Splunk Employee
Splunk Employee

The macro powering the search in the "MB Recevied" Panel is:

sourcetype_metrics_timechart

You can see what is consuming all the time in your search by inspecting the job. Here is what you'll want to do:

  1. Reload the "All Sourcetypes" panel and go on a small coffee break ( not too long of a break or the job details will get cleaned up).

  2. Click on the "Jobs" link in the upper right corner

  3. Click "Inspect" on the entry for "sourcetype_metrics_timechart"

It sounds like the search might not be using Report Acceleration correctly. Do you see a message indicating that sumaries are being used?

It would look something like the following:

DEBUG: [my.host.name] Using summaries for search, 

If Search summaries are being used, then you may have run into a bug in the core product with search acceleration. In that case, I would recommend opening a support case.

araitz
Splunk Employee
Splunk Employee

Not enough information. Can you open a support case please?

0 Karma
Get Updates on the Splunk Community!

Observability | How to Think About Instrumentation Overhead (White Paper)

Novice observability practitioners are often overly obsessed with performance. They might approach ...

Cloud Platform | Get Resiliency in the Cloud Event (Register Now!)

IDC Report: Enterprises Gain Higher Efficiency and Resiliency With Migration to Cloud  Today many enterprises ...

The Great Resilience Quest: 10th Leaderboard Update

The tenth leaderboard update (11.23-12.05) for The Great Resilience Quest is out >> As our brave ...