I am using a summary index to help speed up a search for my dashboard. Prior to using the summary index, the search would take almost 7 minutes to generate stats over 30 days. Using the summary index, it takes about 65 seconds. Which, according to the users, is still too long.
What else can I do to try to speed it up? The summary search is set up to run hourly with earliest set to -4h@h and latest to -3h@h to protect against late arriving files.
The summary search is : "index=foo | sistats count as count sum(filesize) as volume by location, hub, src, direction, priority"
The search that is called from the dashboard (created with sideviewutil) is "index=si-br-summary | stats count as count sum(filesize) as volume by location, hub, src, direction, priority | search location=$location$ | stats sum(eval(if(direction="IN",count,0))) as count_in sum(eval(if(direction="OUT",count,0))) as count_out sum(eval(if(direction="IN", volume, 0))) as volume_in sum(eval(if(direction="OUT", volume, 0))) as volume_out sum(eval(if(direction="IN" AND priority=1, count, 0))) as p1_in sum(eval(if(direction="IN" AND priority=2, count, 0))) as p2_in sum(eval(if(direction="IN" AND priority=3, count, 0))) as p3_in sum(eval(if(direction="OUT" AND priority=1, count, 0))) as p1_out sum(eval(if(direction="OUT" AND priority=2, count, 0))) as p2_out sum(eval(if(direction="OUT" AND priority=3, count, 0))) as p3_out by location,hub, src
The entire search takes 75 seconds to run. So 65 seconds for the initial stats from the summary index and then an additional 10 seconds for the follow on calculations.
The original data comes from a csv file with these fields: location, direction, hub, src, priority, filesize, timestamp. It basically is a listing of files delivered to an ftp server at a location or sent from the ftp server at a location.
I have created the dashboard using sideview utils. It will be used by many different users, each of whom can select the timeframe they want to look at. Most common timeframes will be last 24 hours, last 7 days, last 30 days, but they can choose whatever they want.
The goal is to show a table similar to this:
I thought about summarizing the summary index daily, but then I have to be able to smartly use the daily summary for some parts of the timeframe and the hourly summary for anything in the current day. Is that even possible to do?
... View more