Knowledge Management
Highlighted

Report Acceleration and Summary Index.

Contributor

Hi All,

I have a summary index which summarize information at earliset = -13h@h to latest= -12h@h.

index="blah" | bucket _time span=5min | sistats count sum(bytes) as sum_bytes by hostname status service _time

I want to use the Report Acceleration on this search to fasten the search as i am dealing with huge amount of data. But don't want to screw up my existing summary index searches.

My questions are

  1. Can Report Acceleration and summary index be used at the same time for the above use case.
  2. if that's the case then if the data is summarized already, how will it handle the late arriving logs ?

Thanks alot for all the help and support.

Regards
KK

Highlighted

Re: Report Acceleration and Summary Index.

Contributor

Question 1: I would recommend using one or the other.
By using both you are duplicating the workload. On one hand you create a summary index that sits in your search head and which is maintained by your scheduled search. On the other you are creating another summary index (the accelerated search) in your indexers, which is maintained by the automation built in for acceleration. This automation will run every 10 minutes to update your "accelerated summary index".
The advantage of using summary index is that it's faster to search the data in the summary index if you are on the same search head. There is no loss due to network traffic.
The advantage of using accelerated search is that your data is searchable faster from any search head.

Question 2: it depends on which scheme you choose. If using the "scheduled searches + summary index" solution, if there are late arriving logs you will have data gaps.
On the other hand my understanding is that search acceleration will automatically go back and fill the gaps at the next update cycle.

0 Karma
Highlighted

Re: Report Acceleration and Summary Index.

Esteemed Legend

Yes, Report Acceleration and Summary Index be used at the same time for your use case, but not in the way you mean. You will be getting nothing useful by accelerating your SI-populating searches (think about it: they are already 12-hours+ behind; who cares if they are 12-hours+ minus a few seconds behind?) So what you need to do is leave those searches alone and consider accelerating the searches that use your Summary Index data.

In general, as far as late arriving logs, you never need to worry about late arriving longs when you accelerate reports. The acceleration works by creating additional tsidx instances that allow the searches to more quickly (accurately) rule in/out data buckets for matches and this gets done in the normal process of indexing regardless of when the data arrives.

So when using Summary Index, you very much do have to plan around late arriving logs but you do not have to for Report Acceleration.

0 Karma