Dashboards & Visualizations

Performance issue when using transaction? Dashboard not always loading properly.

the_wolverine
Champion

I have noticed that when I use a transaction-based search in my dashboard panels, the performance is not so great and my panels do not always load (perhaps the search times out)? Other than summarizing these searches, is there anything I can do to improve the performance of my transactional searches?

For example:

index="main" sshd "failed password" root OR "invalid user" NOT "user monitoring" | transaction host maxspan=10m | where linecount>9 | stats count host (over past 24 hours)

Perhaps there is a way to lengthen the timeout (if that's the problem that I'm encountering.)

0 Karma

MartinHarper
Path Finder

Another thing you can do, apart from summarizing the searches, is putting them into a saved search and scheduling them to run every hour. That way the dashboard will display the saved search results near-instantly, rather than kicking off a new instance of the search that could take several minutes to return.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Seems to me you could get similar (not identical) results much faster (and also taking advantage of map-reduce much more effectively, assuming you have multiple indexers) with:

index="main" sshd "failed password" root OR "invalid user" NOT "user monitoring"  
| bucket _time span=10m
| stats count by _time,host
| where count > 9
| stats distinct_count(host)

You'd have to adjust the counts and span from what you have since it will use a fixed rather than sliding time window (e.g., change the threshold to 6, or the span to 20 minutes and the threshold to 15), but the metric is unlikely to be that hard-defined and the performance tradeoff may be worth it.

0 Karma

the_wolverine
Champion

Thanks for your response, Gerard. Your search appears to run faster.

My search, otherwise, works fine for me giving me the data in the format I want and on a sliding window basis. I use this to alert on potential brute force attack on a system and can't limit it to a fixed time window. Is there some way to bypass any timeout I might be hitting in the UI?

0 Karma
Get Updates on the Splunk Community!

Observability Unlocked: Kubernetes Monitoring with Splunk Observability Cloud

  Ready to master Kubernetes and cloud monitoring like the pros?Join Splunk’s Growth Engineering team for an ...

Wrapping Up Cybersecurity Awareness Month

October might be wrapping up, but for Splunk Education, cybersecurity awareness never goes out of season. ...

🌟 From Audit Chaos to Clarity: Welcoming Audit Trail v2

🗣 You Spoke, We Listened  Audit Trail v2 wasn’t written in isolation—it was shaped by your voices.  In ...