Best way of making base search

[N-W]

Hello everyone! 

I need some help with figuring out how to make this base search the best way without hitting the 500.000 limit aswell.

<search id="base_search">
  <query>index=Test | fields orders status countryCode
  <earliest>-60m@m</earliest>
  <latest>now</latest>
</search>

Since i need it to be used with these different searches:

<search base="base_search">
  <query>search | timechart span=1d count(orders) by status</query>
</search>
<search base="base_search">
  <query>search | timechart span=30m count(orders) by status</query>
</search>
<search base="base_search">
  <query>search status=!"Cancelled" | timechart span=1d count(orders) by status</query>
</search>
<search base="base_search">
  <query>search countryCode="SWE" | timechart span=1d count(orders) by status</query>
</search>

Or am I missing something simple? I know base searches needs to be transformative to not hit the cap but how would I do that without making it unable to use the search command for the different things I need later? Like for specific countries etc.?

[gcusello]

Hi @N-W,

At first, 

there's a strange thing in your base search: how can you have a span of 1 day with an earliest time of 60 minutes?

Anyway, the best way to use a base search is using a transforming command (as e.g. timechart or stats, etc...) so in this way you can limit the number of results,  but base searches runs also in the way you used.

Anyway, it's possible to optimize your base search and the others in ths way:

<search id="base_search">
  <query>
     index=Test
     | timechart span=30m count(orders) by status
  <earliest>-7d@d</earliest>
  <latest>now</latest>
</search>

and then use the following searches in panels:

<search base="base_search">
  <query>
     your_search
     | timechart span=1d count(orders) by status
  </query>
</search>
<search base="base_search">
  <query>
  </query>
</search>
<search base="base_search">
  <query>
     | search status=!"Cancelled" 
     | timechart span=1d count(orders) by status
  </query>
</search>
<search>
  <query>
     your_search countryCode="SWE" 
     | timechart span=1d count(orders) by status
  </query>
  <earliest>-7d@d</earliest>
  <latest>now</latest>
</search>

In which the last one is out of the base search because it uses a field different than status.

Ciao.

Giuseppe

[ITWhisperer · Accepted solution]

Since this base search counts by status in 30m buckets, the subsequent searches should sum the counts into daily totals where appropriate. Using stats in the base search keeps the events by time and status giving the subsequent searches useful events to work with.

<search id="base_search">
  <query>
     index=Test
     | bin _time span=30m
     | stats count(orders) as ordercount by _time status
  <earliest>-7d@d</earliest>
  <latest>now</latest>
</search>

and

<search base="base_search">
  <query>
     your_search
     | timechart span=1d sum(ordercount) as dailytotal by status
  </query>
</search>
<search base="base_search">
  <query>
  </query>
</search>
<search base="base_search">
  <query>
     | search status=!"Cancelled" 
     | timechart span=1d sum(ordercount) as dailytotal by status
  </query>
</search>
<search>
  <query>
     your_search countryCode="SWE" 
     | timechart span=1d sum(ordercount) as dailytotal by status
  </query>
  <earliest>-7d@d</earliest>
  <latest>now</latest>
</search>

If you include countryCode in the stats as well, you might be able to use the same base search for that panel too.

<search id="base_search">
  <query>
     index=Test
     | bin _time span=30m
     | stats count(orders) as ordercount by _time status countryCode
  <earliest>-7d@d</earliest>
  <latest>now</latest>
</search>

and

<search base="base_search">
  <query>
     | search countryCode="SWE" 
     | timechart span=1d sum(ordercount) as dailytotal by status
  </query>
  <earliest>-7d@d</earliest>
  <latest>now</latest>
</search>

[N-W]

Thank you this answer made it so that i could also use it for diffrent queries without having to dubble the amount of searches 😃

[N-W]

Thank you i will also try this one 🙂 

[N-W]

Thank you so much for this! I will try this today 😊

[gcusello]

Hi @N-W,

good for you, tell me if I can help you more.

If this answer solves your need, please accept it for the other people of Community

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated 😉