Monitoring Splunk

How do I improve loading time of my dashboard with many scheduled searches running to populate it?

piebob
Motivator

(asking this on behalf of another user)

I have one issue while creating dashboards in my app. To improve loading time in dashboard, I am using lookup created from scheduled searches which are running every 15 mins. Scheduled search is performing some calculations and using timechart command to place daily count in the lookup table. On dashboard time range picker is placed to select data from lookup for particular time range. Now there are multiple panels in dashboard. For each panel there are 5-6 scheduled searches and data is very large. Now it seems to be an issue, since scheduled searches keep on running and degrading the performance. Please suggest.

hcanivel
Explorer

What kind of saved searches are you performing? If they're mostly of using a time chart for a count, you're halfway there. Look into summary indexing if you haven't already: http://docs.splunk.com/Documentation/Splunk/6.0/Knowledge/Usesummaryindexing

See if any of the listed use cases apply to yours.

Definitely check out this if you haven't already:
http://docs.splunk.com/Documentation/Splunk/6.0/AdvancedDev/PostProcess

I'd consider a few other things (mainly taken from the above two references) if you wanted to cobble all these requirements together:

  • How related are these searches? Are they for the most part collecting from the same type of logs?
  • How expensive is each search? Is the purpose of this dashboard to summarize?
  • Do they need to be ad-hoc or real-time? Can you live with hourly updates if they truly are high volume searches/results?
  • How much data did you really want to consume out of this dashboard?
  • And finally/most importantly: is summary indexing ok?

If you can combine a lot of your requirements and using a base search, I think you should be able to achieve the rest. If you want to persist your scheduled searches, I'd recommend summary indexing and using adapted queries in this dashboard to just aggregate from there. Some overhead, but most efficient in the end potentially.

somesoni2
Revered Legend

can you give some example query you are using for panels?

Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!