How to monitor Splunk 6.4 performance?

[arkonner]

Should be possible to determine the resource in use by each search or dashboard (mem, cpu...)

[bchung_splunk]

Yes that's possible,

for ad-hoc searches, you can check out introspection index:

index=_introspection component=PerProcess data.search_props.sid="*" | stats avg(data.pct_cpu) avg(data.mem_used) avg(data.fd_used) avg(written_mb) avg(read_mb) by data.search_props.sid

and since dashboards are composed by searches, you just need to add id for each search by editing xml, e.g:

<dashboard>
  <label>test</label>
  <row>
    <panel>
      <table>
        <search id='dashboardSmall'>
          <query>sourcetype="index_and_load_summary-too_small" | timechart count</query>
        </search>
      </table>
    </panel>
    <panel>
      <table>
        <search id="dashboardResults">
          <query>source="*rt_test_results*" branch=origin-develop commit=* search_type=* rt_concurrent=* rt_router_threads=* rt_blocking=* rt_queue_size=* | stats avg(instantaneous_eps) as metric first(_time) as time by commit search_type rt_concurrent rt_router_threads rt_blocking rt_queue_size | eventstats avg(metric) as metric_avg by search_type rt_concurrent rt_router_threads rt_blocking rt_queue_size | eval ratio=metric/metric_avg</query>
        </search>
      </table>
    </panel>
  </row>
</dashboard>

then once the dashboard is opened, after a while, you can search introspection data for specific dashboard searches (note data.search_props.sid now specified as "*dashboard*" - the search ids of the dashboard😞

index=_introspection component=PerProcess data.search_props.sid="*dashboard*" | stats avg(data.pct_cpu) avg(data.mem_used) avg(data.fd_used) avg(written_mb) avg(read_mb) by data.search_props.sid

Note:

1. Introspection might not catch the resource usages if the search finishes quickly, since introspection checks for resources usages every 10s (but I guess that's okay since quick searches wont have large performance impact)
2. if you get sid like this: admin__admin__search__RMD5624e8baf335d3eaf_1462017607.924, you'll need to shorten your search id.

HTH,
Bill