Dashboards & Visualizations

Performance issue when using transaction? Dashboard not always loading properly.

the_wolverine
Champion

I have noticed that when I use a transaction-based search in my dashboard panels, the performance is not so great and my panels do not always load (perhaps the search times out)? Other than summarizing these searches, is there anything I can do to improve the performance of my transactional searches?

For example:

index="main" sshd "failed password" root OR "invalid user" NOT "user monitoring" | transaction host maxspan=10m | where linecount>9 | stats count host (over past 24 hours)

Perhaps there is a way to lengthen the timeout (if that's the problem that I'm encountering.)

0 Karma

MartinHarper
Path Finder

Another thing you can do, apart from summarizing the searches, is putting them into a saved search and scheduling them to run every hour. That way the dashboard will display the saved search results near-instantly, rather than kicking off a new instance of the search that could take several minutes to return.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Seems to me you could get similar (not identical) results much faster (and also taking advantage of map-reduce much more effectively, assuming you have multiple indexers) with:

index="main" sshd "failed password" root OR "invalid user" NOT "user monitoring"  
| bucket _time span=10m
| stats count by _time,host
| where count > 9
| stats distinct_count(host)

You'd have to adjust the counts and span from what you have since it will use a fixed rather than sliding time window (e.g., change the threshold to 6, or the span to 20 minutes and the threshold to 15), but the metric is unlikely to be that hard-defined and the performance tradeoff may be worth it.

0 Karma

the_wolverine
Champion

Thanks for your response, Gerard. Your search appears to run faster.

My search, otherwise, works fine for me giving me the data in the format I want and on a sliding window basis. I use this to alert on potential brute force attack on a system and can't limit it to a fixed time window. Is there some way to bypass any timeout I might be hitting in the UI?

0 Karma
Get Updates on the Splunk Community!

Index This | Divide 100 by half. What do you get?

November 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Stay Connected: Your Guide to December Tech Talks, Office Hours, and Webinars!

❄️ Celebrate the season with our December lineup of Community Office Hours, Tech Talks, and Webinars! ...

Splunk and Fraud

Watch Now!Watch an insightful webinar where we delve into the innovative approaches to solving fraud using the ...