Dashboards & Visualizations

Performance issue when using transaction? Dashboard not always loading properly.

the_wolverine
Champion

I have noticed that when I use a transaction-based search in my dashboard panels, the performance is not so great and my panels do not always load (perhaps the search times out)? Other than summarizing these searches, is there anything I can do to improve the performance of my transactional searches?

For example:

index="main" sshd "failed password" root OR "invalid user" NOT "user monitoring" | transaction host maxspan=10m | where linecount>9 | stats count host (over past 24 hours)

Perhaps there is a way to lengthen the timeout (if that's the problem that I'm encountering.)

0 Karma

MartinHarper
Path Finder

Another thing you can do, apart from summarizing the searches, is putting them into a saved search and scheduling them to run every hour. That way the dashboard will display the saved search results near-instantly, rather than kicking off a new instance of the search that could take several minutes to return.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Seems to me you could get similar (not identical) results much faster (and also taking advantage of map-reduce much more effectively, assuming you have multiple indexers) with:

index="main" sshd "failed password" root OR "invalid user" NOT "user monitoring"  
| bucket _time span=10m
| stats count by _time,host
| where count > 9
| stats distinct_count(host)

You'd have to adjust the counts and span from what you have since it will use a fixed rather than sliding time window (e.g., change the threshold to 6, or the span to 20 minutes and the threshold to 15), but the metric is unlikely to be that hard-defined and the performance tradeoff may be worth it.

0 Karma

the_wolverine
Champion

Thanks for your response, Gerard. Your search appears to run faster.

My search, otherwise, works fine for me giving me the data in the format I want and on a sliding window basis. I use this to alert on potential brute force attack on a system and can't limit it to a fixed time window. Is there some way to bypass any timeout I might be hitting in the UI?

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Thanks for the Memories: .conf26 Took Learning to New Heights

Thank you, Splunk Community, for making .conf26 in Denver one for the books. From packed Splunk University ...

Best Practices: Splunk auto adjust pipeline queue

When you enable autoAdjustQueue in Splunk, maxSize should be understood as the queue size Splunk starts with ...

Splunk Auto Ingestion Parallel Pipeline Scaling

Why this feature matters Many Splunk environments experience ingestion pressure long before the host is fully ...