Optimization / Post-Process Searches

[willadams]

I am running into a concurrent search / disk quota limit with a custom app I have written. The app sits on my ES search head as the data for an investigation is there and the spot most of our analysts will use. My dashboard has 11 panels in it and I tried to use post-process searches but I think my queries were wrong and hence I deleted my app so I can recreate from scratch (I backed up the config). The search itself looks for a certain set of EventCodes that are produced by Windows to deal with mostly group additions. What I would like to know is how I can optimise my search so that I am using "1" search and not "13" + the ES searches which is grinding once specific account to sit with a permanent "waiting for data".

This was my original script (note that I had a search id as my top search but in the panel I wasn't referencing this as I was missing the search base) in each panel.

<form hideFilters="true" refresh="1800" theme="dark">
  <label>Interesting Events</label>
  <description>This dashboard shows elevation for monitored privilege groups</description>
  <search id="privgroupescalation">
    <query>
      index=windows sourcetype=WinEventLog EventCode IN(1102,4732,4728,4625,4720,4726,4755,4756,4776,4768) | fields *
  </query>
    <earliest>$time_tok.earliest$</earliest>
    <latest>$time_tok.latest$</latest>
  </search>
  <fieldset submitButton="false">
    <input type="time" token="time_tok">
      <label></label>
      <default>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </default>
    </input>
  </fieldset>
  <row>
    <panel>
      <title>Domain Admin Change</title>
      <single>
        <search>
          <query>|  search EventCode="4728" AND Group_Name="Domain Admins" AND action = "success" 
          | timechart dc(Group_Name) span=24h 
          | appendpipe [stats count 
          | where count=0]</query>
          <earliest>$time_tok.earliest$</earliest>
          <latest>$time_tok.latest$</latest>
        </search>
   </single>
</panel>
</row>
</form>

The other panels are pretty much a copy and paste but follow a similar vain, for example

<title>Enterprise Admin Change</title>
      <single>
        <search>
          <query>| search EventCode="4756" AND Group_Name="Enterprise Admins" AND action = "success" 
            | timechart dc(Group_Name) span=24h 
            | appendpipe [stats count 
            | where count=0]</query>

If I fixed this up, my search panel should look something like (add base= in search tag)

<title>Enterprise Admin Change</title>
      <single>
        <search base="privgroupescalation">
          <query>| search EventCode="4756" AND Group_Name="Enterprise Admins" AND action = "success" 
            | timechart dc(Group_Name) span=24h 
            | appendpipe [stats count 
            | where count=0]</query>

Is this the best way to do this so that effectively a dashboard like this uses 1 search but then post-processing it takes the existing data based on the query in each relevant panel? Can I optimize this any further. I considered maybe taking out *fields ** and being prescriptive of what values to pull out.

Would be great to get some thoughts around optimisation here. Thanks.

[gcusello · Accepted solution]

Hi willadams,
your approach is correct, just little hints:

• in the base search don't use fields * but the list of the fields you use in the panels (e.g. in your sample | fields EventCode Group_Name action) , it's a little faster;
• don't repeat earliest and latest options in panels, infact Splunk editor usually gives a warning;
• see if there are other common rows in all panels, e.g. | search EventCode="4756" AND Group_Name="Enterprise Admins" AND action = "success" bring them in base search.

Ciao.
Giuseppe

[willadams]

should I find this should this just be piped into different searches? For example if I had EventCode=4756 with multiple Group names, would it best to do

| search EventCode="4756" AND Group_Name="Enterprise Admins" AND action = "success"
| search EventCode="4756" AND Group_Name="Other Group 1" AND action = "success"
| search EventCode="4756" AND Group_Name="Other Group 2" AND action = "success" etc.

This is looking for single instance events that are then alerted on.

[gcusello]

Hi willadams,
as I said, put in the base search as more as things you can, in your sample you could add EventCode="4756" (Group_Name="Enterprise Admins" OR Group_Name="Other Group *") action = "success" that seem to be common to all panels.
Then use in each panel the not common parts, e.g. in one panel you have to put | search Group_Name="Enterprise Admins" and in another | search Group_Name="Other Group 1" and so on.

Ciao.
Giuseppe

[willadams]

Appreciated Giuseppe.

Thanks

[gcusello]

You're welcome!
Ciao and see next time.
Giuseppe