Splunk Search

Excessive Jobs / Optimized Search

Contributor

I have optimised my search as I can see but I have now run into a problem wherein my search is spawning 39 jobs on each refresh. This is obviously killing resources and causes my dashboard panels to take longer than expected to run and does cause the user limits and search limits to be run. As a consequence I have disabled my app until I can understand why this is happening. This is a full extract of the dashboard that I have created, with a couple of things like specific accounts adjusted here and there. When this dashboard is opened it will generate 39 searches to the Job Queue. It will do the same on any type of refresh. What is happening here or what have I missed? I expected that by using a search id and then in each panel referring back to it using search base I could effectively make this 1 search and then have each panel generate based on the parameters.

<form hideFilters="true" refresh="1800" theme="dark">
  <label> Interesting Events</label>
    <description>This dashboards shows elevation </description>
    <search id="privgrpescalation">
      <query>
        index=windows sourcetype=WinEventLog EventCode IN(1102,4732,4728,4625,4720,4726,4755,4756,4776,4768) | fields dvc_nt_host Group_Domain Group_Name action src_user user _time 
      </query>
        <earliest>$time_tok.earliest$</earliest>
        <latest>$time_tok.latest$</latest>
    </search>
    <fieldset submitButton="false">
      <input type="time" token="time_tok">
        <label></label>
          <default>
            <earliest>-24h@h</earliest>
            <latest>now</latest>
          </default>
      </input>
    </fieldset>
  <row>
    <panel depends="$alwaysHideCSS$">
      <html>
        <style>
          #datimechart_number{width:20% !important;}
          #eatimechart_number{width:20% !important;}
          #datable{width:80% !important;}
          #eatable{width:80% !important;}
        </style>
      </html>
    </panel>
  </row>
  <row>
    <panel id="datimechart_number">
      <title>Domain Admin Change (Last 24 hours)</title>
      <single>
        <search base="privgrpescalation">
          <query>
            | search EventCode="4728" AND Group_Name="Domain Admins" AND action="success"
            | timechart dc(Group_Name) span=24h
            | appendpipe [stats count | where count=0]
          </query>
        </search>
      </single>
    </panel>

    <panel id="datable">
      <title>Domain Admin Change (Last 24 hours)</title>
      <single>
        <search base="privgrpescalation">
          <query>
            | search EventCode="4728" AND Group_Name="Domain Admins" AND action="success"
            | table dvc_nt_host src_user user Group_Domain Group_Name _time
            | sort - _time
          </query>
        </search>
      </single>
    </panel>
  </row>

  <row>
    <panel id="eatimechart_number">
      <title>Enterprise Admin Change (Last 24 hours)</title>
      <single>
        <search base="privgrpescalation">
          <query>
            | search EventCode="4756" AND Group_Name="Enterprise Admins" AND action="success"
            | timechart dc(Group_Name) span=24h
            | appendpipe [stats count | where count=0]
          </query>
        </search>
      </single>
    </panel>

    <panel id="eatable">
      <title>Enterprise Admin Change (Last 24 hours)</title>
      <single>
        <search base="privgrpescalation">
          <query>
            | search EventCode="4756" AND Group_Name="Enterprise Admins" AND action="success"
            | table dvc_nt_host src_user user Group_Domain Group_Name _time
            | sort - _time
          </query>
        </search>
      </single>
    </panel>
  </row>

  <row>
    <panel>
      <title>Privilege Group Additions - group 1</title>
        <single>
          <search base="privgrpescalation">
            <query>
              | search EventCode IN(4755,4756,4728) [|inputlookup privgroup_group1.csv]
              | timechart dc(Group_Name) span=24h
              | appendpipe [stats count | where count=0]
            </query>
          </search>
        </single>
    </panel>
    <panel>
      <title>Privilege Group Additions - group 2</title>
        <single>
          <search base="privgrpescalation">
            <query>
              | search EventCode IN(4755,4756,4728) [|inputlookup privgroup_group2.csv]
              | timechart dc(Group_Name) span=24h
              | appendpipe [stats count | where count=0]
            </query>
          </search>
        </single>
    </panel>
    <panel>
      <title>Privilege Group Additions - group 3</title>
        <single>
          <search base="privgrpescalation">
            <query>
              | search EventCode IN(4755,4756,4728) [|inputlookup privgroup_group3.csv]
              | timechart dc(Group_Name) span=24h
              | appendpipe [stats count | where count=0]
            </query>
          </search>
        </single>
    </panel>
  </row>

  <row>
    <panel>
      <title>Event Logs Cleared</title>
      <single>
        <search base="privgrpescalation">
          <query>
            | search EventCode="1102"
            | timechart dc(user) span=24h
            | appendpipe [stats count | where count=0]
          </query>
        </search>
      </single>
    </panel>

    <panel>
      <title>Internet Access Provided to Account</title>
      <single>
        <search base="privgrpescalation">
          <query>
            | search EventCode="4728" AND Group_Name="internet" AND src_user!="admin"
            | timechart dc(user) span=24h
            | appendpipe [stats count | where count=0]
          </query>
        </search>
      </single>
    </panel>

    <panel>
      <title>User Account Creation</title>
      <single>
        <search base="privgrpescalation">
          <query>
            | search EventCode="4720" src_user!="admin"
            | timechart dc(Group_Name) span=24h
            | appendpipe [stats count | where count=0]
          </query>
        </search>
      </single>
    </panel>

    <panel>
      <title>User Account Deletion</title>
      <single>
        <search base="privgrpescalation">
          <query>
            | search EventCode="4726" src_user!="admin" src_user!="2nd_admin" src_user!="3rd_admin"
            | timechart dc(Group_Name) span=24h
            | appendpipe [stats count | where count=0]
          </query>
        </search>
      </single>
    </panel>
  </row>
</form>

Thanks in advance

0 Karma

SplunkTrust
SplunkTrust

Hi willadams,
your search has a large time period, probably with millions or billions of events , so maybe you could try a different approach. did you explored the choise to use summary index?

To do this, you have to:

  • schedule your main search (e.g. every hour) searching only on the events of the previous hour (e.g. at 11.15 search from 10.00.00 to 10.59.59 earliest=-h@h latest=@h),
  • your search must have as results fields not the row events: so you have to use table or stats or timechart, etc...,
  • save your results in a summary index adding as last row of your search | summary index=my_summary,
  • Then you can run your search (always using post process search) on the summary index that is very quicker than normal search,
  • so you have results of the previous 23 hours from summary index and you have to search on events only on the last few minutes.

e.g. you could try

index=windows sourcetype=WinEventLog EventCode IN(1102,4732,4728,4625,4720,4726,4755,4756,4776,4768)  earliest=-h@h latest=@h
| table _time dvc_nt_host Group_Domain Group_Name action src_user user _time
| collect index=my_summary

Then you could run in base search something like

index=my_summary
| fields _time dvc_nt_host Group_Domain Group_Name action src_user user _time
| append [ search index=windows sourcetype=WinEventLog EventCode IN(1102,4732,4728,4625,4720,4726,4755,4756,4776,4768)  earliest=@h latest=now | fields _time dvc_nt_host Group_Domain Group_Name action src_user user _time ]

Check only if the results of the subsearch are more or less of 50,000 because if the could be more, you have to limit the period of the scheduled search (e.g. every 30 minutes).

In this way you have still all your jobs, but they are very much less heavy for Splunk.

P.S.: Summary indexes don't consume license!)

Ciao.
Giuseppe

0 Karma