topic Re: Use activity in Monitoring Splunk

Use activity

ritchierich — Wed, 30 Sep 2020 04:00:32 GMT

Hi ,
I am trying to print user active from directory

Splunk active/inactive users

<input type="radio" token="active_account">
  <label>Active accounts</label>
  <choice value="*">all</choice>
  <choice value="1">active</choice>
  <choice value="0">inactive</choice>
  <default>1</default>
</input>
<input type="text" token="user_field" searchWhenChanged="true">
  <label>User:</label>
  <default>*</default>
</input>
<input type="text" token="role_field" searchWhenChanged="true">
  <label>Role:</label>
  <default>*</default>
</input>


<panel>
  <table>
    <search>
      <query>| rest /services/authentication/users   | dedup title   | rename title as user | eval firstHit=0  | eval lastHit=0 | eval active=1 | table user, firstHit, lastHit, roles, active  | inputlookup append=true splunk_users | eval user=if(isnull(_key), user, _key)  | stats max(firstHit) as firstHit, max(lastHit) as lastHit, values(roles) as roles, max(active) as active by user | convert timeformat="%Y-%m-%d %H:%M:%S" ctime(firstHit) | convert timeformat="%Y-%m-%d %H:%M:%S" ctime(lastHit)  | eval active=if(active==1, active, 0) | search user="$user_field$" | search active=$active_account$ | search roles="$role_field$"</query>
      <earliest>-15m@m</earliest>
      <latest>now</latest>
    </search>
    <option name="wrap">true</option>
    <option name="rowNumbers">true</option>
    <option name="dataOverlayMode">none</option>
    <option name="drilldown">none</option>
    <option name="count">100</option>
  </table>
</panel>

User/Role/Index Management

<panel>
  <title>Splunk indexes with corresponding roles</title>
  <input type="radio" token="view_field1" searchWhenChanged="true">
    <label>View:</label>
    <choice value="| nomv index">One line</choice>
    <choice value="">Human readable (currently not working)</choice>
    <default>| nomv index</default>
  </input>
  <input type="text" token="role_field1" searchWhenChanged="true">
    <label>Role:</label>
    <default>*</default>
  </input>
  <input type="text" token="index_field1">
    <label>Index:</label>
    <default>*</default>
  </input>
  <table>
    <search>
      <query>| inputlookup  admin_role_indexes

| eval index = mvappend(srchIndexesAllowed, imported_srchIndexesAllowed) | fields role, index $view_field1$ | search role=$role_field1$ | search index=$index_field1$
| dedup role
| rex field=index max_match=200 "(?<idx>\w+)"
| lookup admin_indexes_data_owners index as idx
| stats values(index) as index, values(data_owner) as data_owner by role
-15m@m
now

20
none
none
false
true

<panel>
  <title>Splunk users details</title>
  <input type="radio" token="view_field2" searchWhenChanged="true">
    <label>View:</label>
    <choice value="| nomv index | nomv role">One line</choice>
    <choice value="">Human readable (currently not working)</choice>
    <default>| nomv index | nomv role</default>
  </input>
  <input type="text" token="user_field2" searchWhenChanged="true">
    <label>User:</label>
    <default>*</default>
  </input>
  <input type="text" token="role_field2" searchWhenChanged="true">
    <label>Role:</label>
    <default>*</default>
  </input>
  <input type="text" token="index_field2">
    <label>Index:</label>
    <default>*</default>
  </input>
  <table>
    <search>
      <query>| inputlookup admin_user_index_role | rename roles as role  $view_field2$ | search user=$user_field2$ | search role=$role_field2$ | search index=$index_field2$ | lookup splunk_users _key as user OUTPUT lastHit as last_seen| eval user=if(isnull(_key), user, _key) | convert timeformat="%Y-%m-%d %H:%M:%S" ctime(last_seen) | table user, last_seen, index, role | eval last_seen=if(isnull(last_seen), "never", last_seen)</query>
      <earliest>-15m@m</earliest>
      <latest>now</latest>
    </search>
    <option name="wrap">true</option>
    <option name="rowNumbers">false</option>
    <option name="dataOverlayMode">none</option>
    <option name="drilldown">none</option>
    <option name="count">20</option>
  </table>
</panel>

Re: Use activity

ritchierich — Wed, 05 Feb 2020 07:37:32 GMT

Latest 20nonefalsetrue

Re: Use activity

ritchierich — Wed, 05 Feb 2020 07:38:42 GMT

Latest /search 20 /option none /option>option name="rowNumbers">falsetrue

Re: Use activity

ritchierich — Wed, 05 Feb 2020 07:40:48 GMT

Latest /searchoption name="count"20optionoption name="drill down"noneoptionoption name="rowNumbers"false/optionoption name="wrap"true/option/table/panel/row

Re: Use activity

ritchierich — Wed, 30 Sep 2020 04:03:23 GMT

Unpredictable data volume in Splunk indexes

<input type="radio" token="time_field">
  <label>Splunk data volume alerts:</label>
  <choice value="20">Yesterday</choice>
  <choice value="140">Last 7 days</choice>
  <choice value="600">Last 30 days</choice>
  <default>20</default>
  <initialValue>20</initialValue>
</input>


<panel>
  <html>
  <h1>Information:</h1>
  <div>
    All <font color="#d93f3c">critical</font> alerts are monitored by <a href="https://splunk.analytics.vodafone.com/en-US/app/analytics/alert?s=%2FservicesNS%2Fnobody%2Fanalytics%2Fsaved%2Fsearches%2FSplunk%2520Alert%2520-%2520Detected%2520unpredicted%2520data%2520volume%2520in%2520Splunk%2520indexes" target="_blank">Splunk Alert - Detected unpredicted data volume in Splunk indexes</a> and sent to Operational Intelligence Team.
  </div>
  <div>
    Please also visit <a href="https://splunk.analytics.vodafone.com/en-US/app/analytics/admin_traffic_forecasts_teams_products" target="_blank">Traffic forecasts by teams/products</a> dashboard for more details.
  </div>
</html>
</panel>


<panel>
  <title>Number of indexes with data volume alerts</title>
  <single>
    <search>
      <query>index=splunk_internal_db source=splunk_internals_daily_load

[ search index=splunk_internal_db source=splunk_internals_daily_load
| stats sum(usage) as usageMB by idx
| sort - usageMB
| head 20 | table idx ]
| bucket _time span=1d
| stats sum(eval(usage/1024)) as usage by _time, idx
| streamstats window=400 current=false mean(usage) as median_number by idx
| eval absDev=(abs(usage-median_number))
| streamstats window=400 current=false mean(absDev) as medianAbsDev by idx
| eval lowerBound=abs(median_number-medianAbsDev*3), upperBound=(median_number+medianAbsDev*3)
| eval upperBound=if((median_number/upperBound*100 > 85) OR (median_number/upperBound*100 < 75), median_number+median_number*0.15, upperBound)
| eval lowerBound=if(lowerBound/median_number*100 > 95, median_number*0.90, lowerBound)
| eval isOutlier=if(usage < lowerBound OR usage > upperBound, 1, 0)
| eval priority=case(
isOutlier=1 AND usage>50 AND usage>upperBound, "1. critical",
isOutlier=1 AND usage>20 AND usage<50 AND usage>lowerBound, "2. warning",
isOutlier=1, "3. low"
)
| stats max(isOutlier) as isOutlier, max(usage) as data_volume_GB, values(priority) as priority by _time, idx
| tail $time_field$

| stats dc(idx) as number by priority
| where priority="1. critical"
| table number
-60d@d
@d

block
["0xd93f3c","0xd93f3c"]
[0]
critical
1

index=splunk_internal_db source=splunk_internals_daily_load
[ search index=splunk_internal_db source=splunk_internals_daily_load
| stats sum(usage) as usageMB by idx
| sort - usageMB
| head 20 | table idx ]
| bucket _time span=1d
| stats sum(eval(usage/1024)) as usage by _time, idx
| streamstats window=400 current=false mean(usage) as median_number by idx
| eval absDev=(abs(usage-median_number))
| streamstats window=400 current=false mean(absDev) as medianAbsDev by idx
| eval lowerBound=abs(median_number-medianAbsDev*3), upperBound=(median_number+medianAbsDev*3)
| eval upperBound=if((median_number/upperBound*100 > 85) OR (median_number/upperBound*100 < 75), median_number+median_number*0.15, upperBound)
| eval lowerBound=if(lowerBound/median_number*100 > 95, median_number*0.90, lowerBound)
| eval isOutlier=if(usage < lowerBound OR usage > upperBound, 1, 0)
| eval priority=case(
isOutlier=1 AND usage>50 AND usage>upperBound, "1. critical",
isOutlier=1 AND usage>20 AND usage<50 AND usage>lowerBound, "2. warning",
isOutlier=1, "3. low"
)
| stats max(isOutlier) as isOutlier, max(usage) as data_volume_GB, values(priority) as priority by _time, idx
| tail $time_field$
| stats dc(idx) as number by priority
| where priority="2. warning"
| fillnull value=0 number
| table number
-60d@d
@d

block
["0xf7bc38","0xf7bc38"]
[0]
warning
1

index=splunk_internal_db source=splunk_internals_daily_load
[ search index=splunk_internal_db source=splunk_internals_daily_load
| stats sum(usage) as usageMB by idx
| sort - usageMB
| head 20 | table idx ]
| bucket _time span=1d
| stats sum(eval(usage/1024)) as usage by _time, idx
| streamstats window=400 current=false mean(usage) as median_number by idx
| eval absDev=(abs(usage-median_number))
| streamstats window=400 current=false mean(absDev) as medianAbsDev by idx
| eval lowerBound=abs(median_number-medianAbsDev*3), upperBound=(median_number+medianAbsDev*3)
| eval upperBound=if((median_number/upperBound*100 > 85) OR (median_number/upperBound*100 < 75), median_number+median_number*0.15, upperBound)
| eval lowerBound=if(lowerBound/median_number*100 > 95, median_number*0.90, lowerBound)
| eval isOutlier=if(usage < lowerBound OR usage > upperBound, 1, 0)
| eval priority=case(
isOutlier=1 AND usage>50 AND usage>upperBound, "1. critical",
isOutlier=1 AND usage>20 AND usage<50 AND usage>lowerBound, "2. warning",
isOutlier=1, "3. low"
)
| stats max(isOutlier) as isOutlier, max(usage) as data_volume_GB, values(priority) as priority by _time, idx
| tail $time_field$
| stats dc(idx) as number by priority
| where priority="3. low"
| table number
-60d@d
@d

block
["0x6db7c6","0x6db7c6"]
[0]
low
1

<panel>
  <table>
    <title>Data volume alerts for Top 20 indexes (click for details)</title>
    <search>
      <query>index=splunk_internal_db source=splunk_internals_daily_load

<panel depends="$index_token$">
  <viz type="Splunk_ML_Toolkit.OutliersViz">
    <title>Outlier detection for index=$index_token$ in last 60 days</title>
    <search>
      <query>index=splunk_internal_db source=splunk_internals_daily_load idx=$index_token$

| bucket _time span=1d
| stats sum(eval(usage/1024)) as usage by _time, idx
| streamstats window=20 current=false mean(usage) as median_number by idx
| eval absDev=(abs(usage-median_number))
| streamstats window=20 current=false mean(absDev) as medianAbsDev by idx
| eval lowerBound=abs(median_number-medianAbsDev*3), upperBound=(median_number+medianAbsDev*3)
| eval upperBound=if((median_number/upperBound*100 > 85) OR (median_number/upperBound*100 < 75), median_number+median_number*0.15, upperBound)
| eval lowerBound=if(lowerBound/median_number*100 > 95, median_number*0.90, lowerBound)

| eval isOutlier=if(usage < lowerBound OR usage > upperBound, 1, 0)

| table _time, usage, lowerBound, upperBound, median_number, isOutlier
| rename usage as "data volume [GB]"
-60d@d
@d
1

true

<panel>
  <chart>
    <title>Daily volume by sourcetype for index=$index_token$ in last 10 days</title>
    <search>
      <query>index=_internal tag=LS source=*license_usage.log type=Usage idx=$index_token$ st=*

| bucket _time span=1d
| stats sum(b) as "usage" by _time, st
| eval usage=round(usage/1024/1024/1024,2)
| timechart limit=30 span=1d max(usage) as usage by st
-10d@d
@d

collapsed
GB/day
visible
line
469
progressbar

Re: Use activity

ritchierich — Thu, 06 Feb 2020 03:18:54 GMT

dbinspect status about all indexes

<panel>
  <table>
    <search>
      <query>| dbinspect index=* | stats sum(rawSize) as rawSize, sum(sizeOnDiskMB) as sizeOnDiskMB by index | eval rawSizeGB=round(rawSize/1024/1024/1024,2) | eval sizeOnDiskGB=round(sizeOnDiskMB/1024,2) | fields - rawSize, sizeOnDiskMB | accum rawSizeGB as totalRawSizeGB</query>
      <earliest>0</earliest>
    </search>
    <option name="wrap">true</option>
    <option name="rowNumbers">false</option>
    <option name="dataOverlayMode">none</option>
    <option name="drilldown">none</option>
    <option name="count">100</option>
  </table>
</panel>

Re: Use activity

ritchierich — Thu, 06 Feb 2020 03:24:53 GMT

dbinspect status about all indexes

<panel>
  <table>
    <search>
      <query>| dbinspect index=* | stats sum(rawSize) as rawSize, sum(sizeOnDiskMB) as sizeOnDiskMB by index | eval rawSizeGB=round(rawSize/1024/1024/1024,2) | eval sizeOnDiskGB=round(sizeOnDiskMB/1024,2) | fields - rawSize, sizeOnDiskMB | accum rawSizeGB as totalRawSizeGB</query>
      <earliest>0</earliest>
    </search>
    <option name="wrap">true</option>
    <option name="rowNumbers">false</option>
    <option name="dataOverlayMode">none</option>
    <option name="drilldown">none</option>
    <option name="count">100</option>
  </table>
</panel>

Re: Use activity

ritchierich — Wed, 30 Sep 2020 04:03:26 GMT

Search Performance

<input type="time" token="field1" searchWhenChanged="true">
  <label></label>
  <default>
    <earliest>@d</earliest>
    <latest>now</latest>
  </default>
</input>
<input type="checkbox" token="host_field" searchWhenChanged="true">
  <label>Search Head:</label>
  <choice value="tag=SHC">Search Head Cluster</choice>
  <choice value="tag=TSS_SH">TSS SHC</choice>
  <choice value="host=vgsp26hr">Support/Monitoring SH (vgsp26hr)</choice>
  <choice value="host=splunksh08.ena">SDP SH2 (splunksh08.ena)</choice>
  <choice value="host=now-ena-bac144">SDP DB export (now-ena-bac144)</choice>
  <choice value="host=now-bac806">Legacy SH (now-bac806.prd)</choice>
  <delimiter> OR </delimiter>
  <default>tag=SHC</default>
  <initialValue>tag=SHC</initialValue>
</input>


<panel>
  <chart>
    <title>Number of ad-hoc searches per user (click for details)</title>
    <search>
      <query>index=_audit $host_field$ action=search info=completed search search_id!='*scheduler_*'  savedsearch_name="" OR savedsearch_name="search*"  | top user limit=20</query>
      <earliest>$field1.earliest$</earliest>
      <latest>$field1.latest$</latest>
    </search>
    <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
    <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
    <option name="charting.axisTitleX.visibility">visible</option>
    <option name="charting.axisTitleY.visibility">visible</option>
    <option name="charting.axisTitleY2.visibility">visible</option>
    <option name="charting.axisX.scale">linear</option>
    <option name="charting.axisY.scale">linear</option>
    <option name="charting.axisY2.enabled">0</option>
    <option name="charting.axisY2.scale">inherit</option>
    <option name="charting.chart">bar</option>
    <option name="charting.chart.bubbleMaximumSize">50</option>
    <option name="charting.chart.bubbleMinimumSize">10</option>
    <option name="charting.chart.bubbleSizeBy">area</option>
    <option name="charting.chart.nullValueMode">gaps</option>
    <option name="charting.chart.showDataLabels">all</option>
    <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
    <option name="charting.chart.stackMode">default</option>
    <option name="charting.chart.style">shiny</option>
    <option name="charting.drilldown">all</option>
    <option name="charting.layout.splitSeries">0</option>
    <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
    <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
    <option name="charting.legend.placement">right</option>
    <option name="refresh.display">progressbar</option>
    <drilldown>
      <!-- Use set to specify the new token to be created.
      Use any token from the page or from the click event to produce the value needed. -->
      <set token="drilldown_user_token">$row.user$</set>
      <!-- If we also set the form.sourcetype the input will get updated too
      <set token="form.sourcetype">$row.sourcetype$</set>  -->
    </drilldown>
  </chart>
</panel>
<panel>
  <chart>
    <title>Number of Dashboard vs Typed searches (last 24 hours)</title>
    <search>
      <query>index=_audit $host_field$ action=search info=completed  user=* search_id!="'scheduler*"

| eval type=if(match('search_id',"^\'\d{10}..*'$"),"Typed","Dashboard")
| timechart span=1h c as "Total Searches" by type
-24h@h
now

area
stacked
right
progressbar

<panel>
  <chart>
    <title>Number of ad-hoc searches (last 3 days)</title>
    <search>
      <query>index=_audit $host_field$ action=search info=completed search search_id!='*scheduler_*' search_id!='Summary*' savedsearch_name="" OR savedsearch_name="search*" | chart count by date_hour, date_mday</query>
      <earliest>-2d@d</earliest>
      <latest>now</latest>
    </search>
    <option name="charting.chart">column</option>
    <option name="charting.axisY2.enabled">0</option>
    <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
    <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
    <option name="charting.axisTitleX.visibility">visible</option>
    <option name="charting.axisTitleY.visibility">visible</option>
    <option name="charting.axisTitleY2.visibility">visible</option>
    <option name="charting.axisX.scale">linear</option>
    <option name="charting.axisY.scale">linear</option>
    <option name="charting.axisY2.scale">inherit</option>
    <option name="charting.chart.bubbleMaximumSize">50</option>
    <option name="charting.chart.bubbleMinimumSize">10</option>
    <option name="charting.chart.bubbleSizeBy">area</option>
    <option name="charting.chart.nullValueMode">gaps</option>
    <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
    <option name="charting.chart.stackMode">default</option>
    <option name="charting.chart.style">shiny</option>
    <option name="charting.drilldown">all</option>
    <option name="charting.layout.splitSeries">0</option>
    <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
    <option name="charting.legend.placement">right</option>
    <option name="charting.chart.showDataLabels">none</option>
    <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
  </chart>
</panel>


<panel>
  <table id="detail" depends="$drilldown_user_token$">
    <title>Search details for $drilldown_user_token$</title>
    <search>
      <query>index=_audit $host_field$ action=search  (info=granted OR info=completed) search search_id!='*scheduler_*'  savedsearch_name="" OR savedsearch_name="search*" user=$drilldown_user_token$

| eval span_mins=round((search_lt-search_et)/60,0)
| stats max(_time) as _time, values(search) as user_search, sum(total_run_time) as total_run_time, count(eval(info="completed")) as number, max(span_mins) as span_mins by search_id | where number>0 | stats max(_time) as _time, median(total_run_time) as median_run_time, sum(total_run_time) as total_run_time, sum(number) as number, min(span_mins) as min_span_mins, max(span_mins) as max_span_mins by user_search | table _time, user_search, median_run_time, total_run_time, number, min_span_mins, max_span_mins | sort - _time
$field1.earliest$
$field1.latest$

20
none
none
true
true

<panel>
  <chart>
    <title>Total run time of scheduled searches in hours</title>
    <search>
      <query>index=_internal $host_field$   sourcetype=scheduler status="success"  | timechart limit=30  sum(eval(run_time/3600)) as total_runtime by user</query>
      <earliest>$field1.earliest$</earliest>
      <latest>$field1.latest$</latest>
    </search>
    <selection>
      <set token="selection.earliest">$start$</set>
      <set token="selection.latest">$end$</set>
    </selection>
    <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
    <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
    <option name="charting.axisTitleX.visibility">collapsed</option>
    <option name="charting.axisTitleY.visibility">visible</option>
    <option name="charting.axisTitleY2.visibility">visible</option>
    <option name="charting.axisX.scale">linear</option>
    <option name="charting.axisY.scale">linear</option>
    <option name="charting.axisY2.enabled">0</option>
    <option name="charting.axisY2.scale">inherit</option>
    <option name="charting.chart">area</option>
    <option name="charting.chart.bubbleMaximumSize">50</option>
    <option name="charting.chart.bubbleMinimumSize">10</option>
    <option name="charting.chart.bubbleSizeBy">area</option>
    <option name="charting.chart.nullValueMode">zero</option>
    <option name="charting.chart.showDataLabels">none</option>
    <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
    <option name="charting.chart.stackMode">stacked</option>
    <option name="charting.chart.style">shiny</option>
    <option name="charting.drilldown">all</option>
    <option name="charting.layout.splitSeries">0</option>
    <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
    <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
    <option name="charting.legend.placement">right</option>
  </chart>
</panel>


<panel>
  <input type="text" token="user_field">
    <label>User:</label>
    <default>admin</default>
  </input>
  <input type="text" token="pattern_field">
    <label>Pattern:</label>
    <default>*</default>
  </input>
  <table>
    <title>Scheduled searches per user with status=success</title>
    <search>
      <query>index=_internal $host_field$   sourcetype=scheduler status="success" user="$user_field$"  $pattern_field$ | stats max(_time) as _time, mean(run_time) as mean_run_time_seconds, max(run_time) as max_run_time_seconds, count as number_of_jobs, sum(run_time) as total_run_time_seconds by savedsearch_name, user | eval total_run_time_hours=round(total_run_time_seconds/3600,2) | table savedsearch_name, user, _time, mean_run_time_seconds, max_run_time_seconds, number_of_jobs, total_run_time_seconds, total_run_time_hours</query>
      <earliest>$field1.earliest$</earliest>
      <latest>$field1.latest$</latest>
    </search>
    <option name="wrap">true</option>
    <option name="rowNumbers">false</option>
    <option name="dataOverlayMode">none</option>
    <option name="list.drilldown">full</option>
    <option name="list.wrap">1</option>
    <option name="maxLines">5</option>
    <option name="raw.drilldown">full</option>
    <option name="table.drilldown">all</option>
    <option name="table.wrap">1</option>
    <option name="type">list</option>
    <option name="drilldown">cell</option>
    <option name="count">30</option>
  </table>
</panel>


<panel>
  <table>
    <title>Heatmap with scheduled searches (status=*)</title>
    <search>
      <query>index=_internal $host_field$ sourcetype=scheduler status=*| eval alert_actions = if(isnull(alert_actions) OR alert_actions == "", "none", alert_actions) | stats count values(reason) as reasons, values(concurrency_limit) as concurrency_limits  by user, host, status  | sort - count | eventstats sum(count) AS total  | eval percent = round(count / total * 100, 2)." %" | fields - total | rename user as User, count as Count, percent as "Percent of Total"</query>
      <earliest>$field1.earliest$</earliest>
      <latest>$field1.latest$</latest>
    </search>
    <option name="count">30</option>
    <option name="dataOverlayMode">heatmap</option>
    <option name="drilldown">cell</option>
    <option name="refresh.display">progressbar</option>
    <option name="rowNumbers">false</option>
    <option name="wrap">true</option>
    <format type="color" field="status">
      <colorPalette type="map">{"continued":#F8BE34,"skipped":#F1813F}</colorPalette>
    </format>
  </table>
</panel>


<panel>
  <chart>
    <title>Total run time and number of scheduled searches (today/yesterday)</title>
    <search>
      <query>index=_internal $host_field$   sourcetype=scheduler status="success"  | timechart span=1h  sum(eval(run_time/3600)) as total_runtime, count as total_jobs

| eval total_runtime=round(total_runtime,2)
-1d@d
@h

ellipsisNone
0
collapsed
visible
visible
linear
linear
1
inherit
column
50
10
area
gaps
total_jobs
none
0.01
default
shiny
all
0
0
ellipsisMiddle
bottom
progressbar

Frequency of hitting search concurrency limits

index=_internal tag=SHC OR tag=TSS_SH OR host=vgsp26hr sourcetype=scheduler status=continued OR status=skipped "The maximum number of concurrent historical scheduled searches on this cluster has been reached"
| timechart span=1m max(concurrency_limit) by host
$field1.earliest$
$field1.latest$
1

ellipsisNone
0
visible
visible
visible
none
linear
none
linear
none
0
inherit
line
50
10
area
gaps
none
0.01
default
shiny
none
0
0
ellipsisMiddle
standard
right
2
progressbar
0
1
medium

<panel>
  <chart>
    <title>Search concurrency</title>
    <search>
      <query>index=_internal host=splunksh* OR tag=SHC OR host=vgsp26hr OR $host_field$ source="*metrics.log" "system total" search_concurrency | timechart max(active_hist_searches) as active_searches by host</query>
      <earliest>$field1.earliest$</earliest>
      <latest>$field1.latest$</latest>
    </search>
    <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
    <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
    <option name="charting.axisTitleX.visibility">collapsed</option>
    <option name="charting.axisTitleY.text">Number of running searches</option>
    <option name="charting.axisTitleY.visibility">visible</option>
    <option name="charting.axisTitleY2.visibility">visible</option>
    <option name="charting.axisX.scale">linear</option>
    <option name="charting.axisY.scale">linear</option>
    <option name="charting.axisY2.enabled">0</option>
    <option name="charting.axisY2.scale">inherit</option>
    <option name="charting.chart">area</option>
    <option name="charting.chart.bubbleMaximumSize">50</option>
    <option name="charting.chart.bubbleMinimumSize">10</option>
    <option name="charting.chart.bubbleSizeBy">area</option>
    <option name="charting.chart.nullValueMode">connect</option>
    <option name="charting.chart.showDataLabels">none</option>
    <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
    <option name="charting.chart.stackMode">stacked</option>
    <option name="charting.chart.style">shiny</option>
    <option name="charting.drilldown">all</option>
    <option name="charting.layout.splitSeries">0</option>
    <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
    <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
    <option name="charting.legend.placement">bottom</option>
    <option name="refresh.display">progressbar</option>
  </chart>
</panel>


<panel>
  <title>Top 50 memory consuming searches</title>
  <table>
    <search>
      <query>index=_introspection $host_field$ sourcetype=splunk_resource_usage data.search_props.sid::*

| rename data.elapsed as elapsed, data.mem_used as mem_used, data.search_props.sid as sid, data.search_props.label as label, data.search_props.provenance as provenance, data.search_props.type as type, data.search_props.mode as mode, data.search_props.app as app, data.search_props.user as user
| fillnull value=missing label
| stats max(elapsed) as runtime max(mem_used) as mem_used earliest(_time) as _time by sid, label, provenance, type, mode, app, host, user
| eval mem_used = round(mem_used, 2)
| sort 50 - mem_used
| fields - day, hour, minute, second
| eval _time = strftime(_time,"%+")
| table label, mem_used, app, user, *
| rename sid as SID, label as "Search Name", provenance AS Provenance, type as Type, mode as Mode, app as App, search_head as "Search Head", user as User, mem_used as "Memory Usage (MB)", _time as Started, runtime as Runtime
$field1.earliest$
$field1.latest$

10
none
progressbar

[#65A637,#F7BC38,#D93F3C]
500,5000

Re: Use activity

ritchierich — Wed, 30 Sep 2020 04:03:38 GMT

index="_internal" source="*license_usage.log" type=Usage | bin _time span=1d | stats sum(b) AS bytes by _time,idx | eval DailyGB=bytes/1024/1024/1024 | timechart sum(DailyGB) by idx span=1d

Re: Use activity

ritchierich — Wed, 30 Sep 2020 04:03:59 GMT

Monitor license usage
Use the foreach command to monitor license usage.

First run the following search on the license master to return the daily license usage per sourcetype in bytes:

index=_internal source=*license_usage.log type!="*Summary" earliest=-30d
| timechart span=1d sum(b) AS daily_bytes by st

Use the foreach command to calculate the daily license usage in gigabytes for each field:

index=_internal source=*license_usage.log type!="*Summary" earliest=-30d
| timechart span=1d sum(b) AS daily_bytes by st
| foreach * [eval <>='<>'/1024/1024/1024]