Should be possible to determine the resource in use by each search or dashboard (mem, cpu...)
Yes that's possible,
for ad-hoc searches, you can check out introspection index:
index=_introspection component=PerProcess data.search_props.sid="*" | stats avg(data.pct_cpu) avg(data.mem_used) avg(data.fd_used) avg(written_mb) avg(read_mb) by data.search_props.sid
and since dashboards are composed by searches, you just need to add id for each search by editing xml, e.g:
<dashboard>
<label>test</label>
<row>
<panel>
<table>
<search id='dashboardSmall'>
<query>sourcetype="index_and_load_summary-too_small" | timechart count</query>
</search>
</table>
</panel>
<panel>
<table>
<search id="dashboardResults">
<query>source="*rt_test_results*" branch=origin-develop commit=* search_type=* rt_concurrent=* rt_router_threads=* rt_blocking=* rt_queue_size=* | stats avg(instantaneous_eps) as metric first(_time) as time by commit search_type rt_concurrent rt_router_threads rt_blocking rt_queue_size | eventstats avg(metric) as metric_avg by search_type rt_concurrent rt_router_threads rt_blocking rt_queue_size | eval ratio=metric/metric_avg</query>
</search>
</table>
</panel>
</row>
</dashboard>
then once the dashboard is opened, after a while, you can search introspection data for specific dashboard searches (note data.search_props.sid now specified as "*dashboard*" - the search ids of the dashboard
😞
index=_introspection component=PerProcess data.search_props.sid="*dashboard*" | stats avg(data.pct_cpu) avg(data.mem_used) avg(data.fd_used) avg(written_mb) avg(read_mb) by data.search_props.sid
Note:
admin__admin__search__RMD5624e8baf335d3eaf_1462017607.924
, you'll need to shorten your search id.HTH,
Bill