Splunk Search

Why does the dashboard search break, become limited, or run extremely slow but searching through the search app works fine?

SLoBello
Explorer

Lets say I have a search:
((value1 OR value_*) OR (status=404 OR status=500 OR status=503)) (index="main" OR index="secondary" )

(practically identical to the actual search used; in terms of parenthesis, ORs, etc)

Through the regular search app, this works fine and finishes up pretty quick (even with indexes that have a million or more events within 24 hours)

On the Dashboard one of three things happen:

  • Disk usage limit reached: 100mb (I don't want to change this, just to understand why it is happening so often in the dashboard)

  • Another Warning: These results may be truncated. This visualization is configured to display a maximum of 10000 results per series, and that limit has been reached. (only happens when I search indexes with too long a time range... this has only occurred when I accidentally tick a large time range)

  • The search runs slow as if in Verbose mode. (but the config files haven't been touched, and other searches run fairly quick)

I'm fairly new to Splunk, so there may be some (or many...) technicalities, practices, and/or techniques that I'm not too familiar with.
Any detailed insight would be greatly appreciated!

0 Karma
1 Solution

HiroshiSatoh
Champion
0 Karma

HiroshiSatoh
Champion
0 Karma

SLoBello
Explorer

I'm accepting this as an answer to close the question. However, I appreciate the link you provided. Helps me understand the quotas significantly better!

0 Karma

elliotproebstel
Champion

It's pretty difficult to tell what's going on underneath the hood on the dashboard. Do you have access to the dashboard XML code? If you could share that, we would have a better shot at helping you troubleshoot.

0 Karma

SLoBello
Explorer

I figured it out just now:
I had a field search for itself without any conditions narrowing it down. So it searched for every event within that single index along with the other search. It was a post-processing search that i didn't join correctly. Everything is peachy now! Thank you so much for the prompt reply, I really appreciate it!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...