Dashboards & Visualizations

Performance issue when using transaction? Dashboard not always loading properly.

the_wolverine
Champion

I have noticed that when I use a transaction-based search in my dashboard panels, the performance is not so great and my panels do not always load (perhaps the search times out)? Other than summarizing these searches, is there anything I can do to improve the performance of my transactional searches?

For example:

index="main" sshd "failed password" root OR "invalid user" NOT "user monitoring" | transaction host maxspan=10m | where linecount>9 | stats count host (over past 24 hours)

Perhaps there is a way to lengthen the timeout (if that's the problem that I'm encountering.)

0 Karma

MartinHarper
Path Finder

Another thing you can do, apart from summarizing the searches, is putting them into a saved search and scheduling them to run every hour. That way the dashboard will display the saved search results near-instantly, rather than kicking off a new instance of the search that could take several minutes to return.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Seems to me you could get similar (not identical) results much faster (and also taking advantage of map-reduce much more effectively, assuming you have multiple indexers) with:

index="main" sshd "failed password" root OR "invalid user" NOT "user monitoring"  
| bucket _time span=10m
| stats count by _time,host
| where count > 9
| stats distinct_count(host)

You'd have to adjust the counts and span from what you have since it will use a fixed rather than sliding time window (e.g., change the threshold to 6, or the span to 20 minutes and the threshold to 15), but the metric is unlikely to be that hard-defined and the performance tradeoff may be worth it.

0 Karma

the_wolverine
Champion

Thanks for your response, Gerard. Your search appears to run faster.

My search, otherwise, works fine for me giving me the data in the format I want and on a sliding window basis. I use this to alert on potential brute force attack on a system and can't limit it to a fixed time window. Is there some way to bypass any timeout I might be hitting in the UI?

0 Karma
Get Updates on the Splunk Community!

Accelerating Observability as Code with the Splunk AI Assistant

We’ve seen in previous posts what Observability as Code (OaC) is and how it’s now essential for managing ...

Integrating Splunk Search API and Quarto to Create Reproducible Investigation ...

 Splunk is More Than Just the Web Console For Digital Forensics and Incident Response (DFIR) practitioners, ...

Congratulations to the 2025-2026 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...