Base Searches and postprocess

[raphgoncalves]

Hi!

I have a dashboard with 4 panels. I use a base search "baseSearch1" and two post process searches based on my baseSearch1 :

• baseSearch1 -> countSearch

• baseSearch1 -> tableSearch

  
  
  | loadjob savedsearch="xx:yy:REPORT"
  $optionalSearch$
  | search element = $element$
  | lookup lookup.csv ...
  | eval delta = now() - _time
  | eval STATUS = if(STATUS = "InProgress" AND delta > duration, "Warning", STATUS)
  
  $date_filter.earliest$
  $date_filter.latest$
  
  
  | append [
  | loadjob savedsearch="xx:yy:SUB_REPORT"
  | search [search
  index=my_index AND
  
  element = $element$ source IN (source1, source2)
  latest=$date_filter.latest$
  earliest=$date_filter.earliest$
  | fields element
  ] $optionalSearch$
  | search element = $element$
  ]
  | stats last(*) as * by element = $element$
  
  
  
  | search [
  search index=my_index AND
  element = $element$
  source IN (source1,source2)
  latest=$date_filter.latest$
  earliest=$date_filter.earliest$
  | fields element
  ] $optionalSearch$
  | search element = $element$
  ]
  | chart first(STATUS) by flow, element useother=f limit=0
  | sort flow
  | transpose 10 column_name=element header_field=flow
  | join element [
  | loadjob savedsearch="xx:yy:REPORT_DETAIL"
  ]
  
  

Then my panels use these two searches to make some last transformations. When doing so, it turns out that my search lasts much longer than without using base searches (4x longer). It stays stuck on "Waiting for data" for about 30 seconds...

Would someone be able to explain how this is possible ? What could cause this problem ?

Thank you by advance for your help !

[DalJeanis · Accepted solution]

First, please review line 27. by element = $element$ is not syntactically correct.

Second, please explain the format and usage of $optionalSearch$, since we're having trouble imagining a syntax that would work and be useful on line 4 that would also work and be useful on lines 24 and 40.

Third, lines 25 and 41 appear redundant with the searches they follow.

Fourth, overall this architecture seems unnecessarily complex, but without more information on your underlying report formats and data elements, it's hard to be certain or suggest alternatives. We suspect that your innermost subsearch, duplicated on lines 18 and 33, is the underlying issue. Given that there is only one field returned, which must necessarily match the $element$ selected, we are assuming that the detail of the record format and the intention of the subsearches will be needed to verify anything we might suggest.

If that innermost subsearch is intended to return the _time or other internal fields, we suggest replacing fields with table so that the retained fields are clearly noted. On the other hand, if the subsearches are only intended to eliminate the reports when there is no relevant event in the timeframe selected, and to do so by passing the (redundant) "element" value, then we suggest removing that subsearch to a separate search, and using the <done> feature to <set> the value of a token to be used in this search.

[raphgoncalves]

From your answers, I've removed the subsearch that you suspected to be causing the performance issue as it is not necessary in a long term point of view. The dashboard response time lasts now 2/3 seconds ! Many thanks for your help !

[raphgoncalves]

Thank you for your answer. Here are some contextual elements.

My dashboard is following up a process flow which is captured within about ten sources. The id that is going through this flow is "element". My inner subsearch aims to filter events that have occured at the begining of my flow, in order to prevent unrelevant data to be displayed when launching my dashboard.

First point : is a mistake while trying to anonymize my data

Second point : $optionalSearch$ is meant to filter events based on user selections (combining several of its inputs) i.e optionalSearch = | search field1=foo field2 IN (foo*, bar*)

Thrid point : Lines 25 and 41 are indeed redundant, thank you for pointing that out !

Fourth : my reports return "stats'ed" events. The inner subsearch returns a list of element. It is used to filter only events that have elements present in 2 given source, which correspond to the beginning of my flow. Maybe this is useless since it's meant to remove undesired "element" when starting to capture data.

[DalJeanis]

It's best to mark your code with the code button (101 010) or surround it by grave accents (the one over the tilde ~ key) or put it on a line with at least four spaces in front of it, so that HTML-like aspects will not be stripped out by the interface.

[raphgoncalves]

Thank you for your feedback, I've updated the post accordingly

[kmaron]

could you share your search (with sensitive data masked)? it would be much easier to help if we could see it.

[raphgoncalves]

Sorry for the delay... Christmas holidays !
Happy new year everyone ! I've added my searches information in the question