Vienna User Group

Vienna Splunk Meetup 2021 Q1

peter_krammer
Communicator

Per Field Throttling

Throttle alert per field from the Alert settings page:

splunk_alert_throttle.png

Alert Throttle for 7 days per error_name

 

| inputlookup alert_lookup_example.csv append=true
| eval now=now(), alerted_time = coalesce(alerted_time, now)
| stats min(alerted_time) as alerted_time, max(now) as now by error_name
| eval throwout_threshhold = now - (3600 * 24 * 7)
| where alerted_time > throwout_threshhold 
| outputlookup alert_lookup_example.csv
| where alerted_time = now

 

Errors seen within last 7 days, but not in last 7-14 days

 

| makeresults count=100
| streamstats count as day
| eval error_name=case(day = 15, "error0", day = 8, "error1", day = 6, "error2", day % 3 = 0, "error3", day % 2 = 0, "error4")
| eval _time = _time - (day * 3600 * 24)
| where isnotnull(error_name)
    
| inputlookup alert_lookup_example.csv append=true
| stats min(_time) as _time by error_name
| where _time >= now() - (3600 * 24 * 14)
| outputlookup alert_lookup_example.csv
| where _time >= now() - (3600 * 24 * 7)

 

Dokumentation:

Time Filter on Lookups

Filter a lookup containing _time with a timepicker

 

| inputlookup mylookup.csv WHERE [
  | makeresults count=1
  | addinfo
  | eval search="( ( _time >= " . info_min_time . " ) AND ( _time < " . replace(info_max_time, "\+Infinity", "2147483647") . " ) )"
  | table search
]

 

Dokumentation:

Fill holes in timecharts

Fill last 14 days with some events to make sure timechart is continuous.

If you need a count, use eval count=1 for real events and count=0 for created events and make sum(count) as count to get real event count.

 

index=_internal date_wday=tuesday
| eval count=1
| append [ 
    | gentimes start=-14 end=-1 increment=1d 
    | eval _time=starttime, count=0
    | fields _time, count
]
| timechart span=1d sum(count) as count
<base_search>
| timechart span=1d count
| append [ 
    | gentimes start=-14 end=-1 increment=1d 
    | eval _time=starttime, count=0
    | fields _time, count
]
| timechart span=1d sum(count) as count

 

Dokumentation:

Peters Sample Dashboard

<too big, see attachments>

Timewrap

Using Timewrap makes the timeline specific to your searchframe

 

index=_internal group="per_sourcetype_thruput" sourcetype=splunkd earliest="-2d@d" latest="@h"
| timechart span=1h avg(eps)
| timewrap series=exact time_format="%d-%m-%Y" 1d

 

Using alternative that makes the timeframe to start at 00:00 and end at 23:59

 

index=_internal group="per_sourcetype_thruput" sourcetype=splunkd earliest="-2d@d" latest="@h"
| bin _time span=1h
| addinfo
| eval date = "avg(eps)_" . strftime(_time,"%d-%m-%Y"), _time = strptime(strftime(info_max_time, "%d-%m-%Y") .  " " . strftime(_time,"%H:%M:%S"), "%d-%m-%Y %H:%M:%S")
| chart avg(eps) over _time by date

 

Trick how to get the previous result with using timewrap

 

index=_internal group="per_sourcetype_thruput" sourcetype=splunkd earliest="-2d@d" latest="+1d@d"
| timechart span=1h avg(eps)
| timewrap series=exact time_format="%d-%m-%Y" 1d

 

Dokumentation:

Reltime

Example Dashboard

 

<form>
  <label>Reltime</label>
  <search>
    <query>
      | makeresults 
      | addinfo 
      | eval _time = info_min_time | reltime | rename reltime as reltime1
      | eval _time = info_max_time | reltime | rename reltime as reltime2
    </query>
    <earliest>$time_token.earliest$</earliest>
    <latest>$time_token.latest$</latest>
    <done>
      <set token="reltime1">$result.reltime1$</set>
      <set token="reltime2">$result.reltime2$</set>
    </done>
  </search>
  <fieldset submitButton="false">
    <input type="time" token="time_token">
      <label></label>
      <default>
        <earliest>-7d@h</earliest>
        <latest>now</latest>
      </default>
    </input>
  </fieldset>
  <row>
    <panel>
      <title>Events between $reltime1$ and $reltime2$</title>
      <table>
        <search>
          <query>
            | makeresults 
            | addinfo 
            | eval _time = info_min_time | reltime | rename reltime as reltime1
            | eval _time = info_max_time | reltime | rename reltime as reltime2
          </query>
          <earliest>$time_token.earliest$</earliest>
          <latest>$time_token.latest$</latest>
        </search>
        <option name="drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </table>
    </panel>
  </row>
</form>

 

 

 

1 Comment
peter_krammer
Communicator

Timegap Dashboard

<form theme="light">
  <label>Timegap Example</label>
  <search>
    <query>| makeresults | addinfo | eval starttime = strftime(info_min_time, "%m/%d/%Y:%H:%M:%S"), endtime = strftime(info_max_time, "%m/%d/%Y:%H:%M:%S")</query>
    <earliest>$gapfill_time.earliest$</earliest>
    <latest>$gapfill_time.latest$</latest>
    <done>
      <set token="starttime">$result.starttime$</set>
      <set token="endtime">$result.endtime$</set>
    </done>
  </search>
  <fieldset submitButton="false"></fieldset>
  <row>
    <panel>
      <title>Always Data</title>
      <chart>
        <search>
          <query>index=_internal date_wday="thursday" 
| timechart span=1d count</query>
          <earliest>-14d@d</earliest>
          <latest>now</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="charting.axisTitleX.visibility">collapsed</option>
        <option name="charting.chart">line</option>
        <option name="charting.drilldown">none</option>
        <option name="charting.legend.mode">standard</option>
        <option name="charting.legend.placement">none</option>
      </chart>
    </panel>
    <panel>
      <title>No Data</title>
      <chart>
        <search>
          <query>index=_internal date_wday="thursday" 
| timechart span=1d count</query>
          <earliest>-4d@d</earliest>
          <latest>now</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="charting.axisTitleX.visibility">collapsed</option>
        <option name="charting.chart">line</option>
        <option name="charting.drilldown">none</option>
        <option name="charting.legend.mode">standard</option>
        <option name="charting.legend.placement">none</option>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <title>Always Data</title>
      <single>
        <search>
          <query>index=_internal date_wday="thursday" 
| timechart span=1d count</query>
          <earliest>-14d@d</earliest>
          <latest>now</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="drilldown">none</option>
      </single>
    </panel>
    <panel>
      <title>No Data</title>
      <single>
        <search>
          <query>index=_internal date_wday="thursday" 
| timechart span=1d count</query>
          <earliest>-4d@d</earliest>
          <latest>now</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="drilldown">none</option>
      </single>
    </panel>
  </row>
  <row>
    <panel>
      <title>Gapfill - short time</title>
      <chart>
        <search>
          <query>index=_internal date_wday="thursday" 
| fields
| eval count=1
| append [ 
    | gentimes start=-14 end=-1 increment=1d 
    | eval _time=starttime, count=0
    | fields _time, count
]
| timechart span=1d sum(count) as count</query>
          <earliest>-4d@d</earliest>
          <latest>now</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="charting.axisTitleX.visibility">collapsed</option>
        <option name="charting.chart">line</option>
        <option name="charting.drilldown">none</option>
        <option name="charting.legend.mode">standard</option>
        <option name="charting.legend.placement">none</option>
      </chart>
    </panel>
    <panel>
      <title>Gapfill - correct time</title>
      <chart>
        <search>
          <query>index=_internal date_wday="thursday" 
| fields
| eval count=1
| append [ 
    | gentimes start=-14 end=-1 increment=1d 
    | eval _time=starttime, count=0
    | fields _time, count
]
| timechart span=1d sum(count) as count</query>
          <earliest>-14d@d</earliest>
          <latest>now</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="charting.axisTitleX.visibility">collapsed</option>
        <option name="charting.chart">line</option>
        <option name="charting.drilldown">none</option>
        <option name="charting.legend.mode">standard</option>
        <option name="charting.legend.placement">none</option>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <title>Gapfill - short time</title>
      <single>
        <search>
          <query>index=_internal date_wday="thursday" 
| fields
| eval count=1
| append [ 
    | gentimes start=-14 end=-1 increment=1d 
    | eval _time=starttime, count=0
    | fields _time, count
]
| timechart span=1d sum(count) as count</query>
          <earliest>-4d@d</earliest>
          <latest>now</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="drilldown">none</option>
      </single>
    </panel>
    <panel>
      <title>Gapfill - correct time</title>
      <single>
        <search>
          <query>index=_internal date_wday="thursday" 
| fields
| eval count=1
| append [ 
    | gentimes start=-14 end=-1 increment=1d 
    | eval _time=starttime, count=0
    | fields _time, count
]
| timechart span=1d sum(count) as count</query>
          <earliest>-14d@d</earliest>
          <latest>now</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="drilldown">none</option>
      </single>
    </panel>
  </row>
  <row>
    <panel>
      <title>Gapfill - timepicker wrong</title>
      <input type="time" token="gapfill_time">
        <label>gapfill_time</label>
        <default>
          <earliest>-14d@d</earliest>
          <latest>now</latest>
        </default>
      </input>
      <chart>
        <search>
          <query>index=_internal date_wday="thursday" 
| fields
| eval count=1
| append [ 
    | gentimes start=-14 end=-1 increment=1d 
    | eval _time=starttime, count=0
    | fields _time, count
]
| timechart span=1d sum(count) as count</query>
          <earliest>$gapfill_time.earliest$</earliest>
          <latest>$gapfill_time.latest$</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="charting.axisTitleX.visibility">collapsed</option>
        <option name="charting.chart">line</option>
        <option name="charting.drilldown">none</option>
        <option name="charting.legend.mode">standard</option>
        <option name="charting.legend.placement">none</option>
      </chart>
    </panel>
    <panel>
      <title>Gapfill - timepicker correct</title>
      <chart>
        <search>
          <query>index=_internal date_wday="thursday" 
| fields
| eval count=1
| append [ 
    | gentimes start=$starttime$ end=$endtime$  increment=1d 
    | eval _time=starttime, count=0
    | fields _time, count
]
| timechart span=1d sum(count) as count</query>
          <earliest>$gapfill_time.earliest$</earliest>
          <latest>$gapfill_time.latest$</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="charting.axisTitleX.visibility">collapsed</option>
        <option name="charting.chart">line</option>
        <option name="charting.drilldown">none</option>
        <option name="charting.legend.mode">standard</option>
        <option name="charting.legend.placement">none</option>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <title>Gapfill - timepicker wrong</title>
      <single>
        <search>
          <query>index=_internal date_wday="thursday" 
| fields
| eval count=1
| append [ 
    | gentimes start=-14 end=-1 increment=1d 
    | eval _time=starttime, count=0
    | fields _time, count
]
| timechart span=1d sum(count) as count</query>
          <earliest>$gapfill_time.earliest$</earliest>
          <latest>$gapfill_time.latest$</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="drilldown">none</option>
      </single>
    </panel>
    <panel>
      <title>Gapfill - timepicker correct</title>
      <single>
        <search>
          <query>index=_internal date_wday="thursday" 
| fields
| eval count=1
| append [ 
    | gentimes start=$starttime$ end=$endtime$  increment=1d 
    | eval _time=starttime, count=0
    | fields _time, count
]
| timechart span=1d sum(count) as count</query>
          <earliest>$gapfill_time.earliest$</earliest>
          <latest>$gapfill_time.latest$</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="drilldown">none</option>
      </single>
    </panel>
  </row>
</form>