Splunk Search

How to monitor Splunk 6.4 performance?

arkonner
Path Finder

Should be possible to determine the resource in use by each search or dashboard (mem, cpu...)

bchung_splunk
Splunk Employee
Splunk Employee

Yes that's possible,

for ad-hoc searches, you can check out introspection index:

index=_introspection component=PerProcess data.search_props.sid="*" | stats avg(data.pct_cpu) avg(data.mem_used) avg(data.fd_used) avg(written_mb) avg(read_mb) by data.search_props.sid

and since dashboards are composed by searches, you just need to add id for each search by editing xml, e.g:

<dashboard>
  <label>test</label>
  <row>
    <panel>
      <table>
        <search id='dashboardSmall'>
          <query>sourcetype="index_and_load_summary-too_small" | timechart count</query>
        </search>
      </table>
    </panel>
    <panel>
      <table>
        <search id="dashboardResults">
          <query>source="*rt_test_results*" branch=origin-develop commit=* search_type=* rt_concurrent=* rt_router_threads=* rt_blocking=* rt_queue_size=* | stats avg(instantaneous_eps) as metric first(_time) as time by commit search_type rt_concurrent rt_router_threads rt_blocking rt_queue_size | eventstats avg(metric) as metric_avg by search_type rt_concurrent rt_router_threads rt_blocking rt_queue_size | eval ratio=metric/metric_avg</query>
        </search>
      </table>
    </panel>
  </row>
</dashboard>

then once the dashboard is opened, after a while, you can search introspection data for specific dashboard searches (note data.search_props.sid now specified as "*dashboard*" - the search ids of the dashboard😞

index=_introspection component=PerProcess data.search_props.sid="*dashboard*" | stats avg(data.pct_cpu) avg(data.mem_used) avg(data.fd_used) avg(written_mb) avg(read_mb) by data.search_props.sid

Note:

  1. Introspection might not catch the resource usages if the search finishes quickly, since introspection checks for resources usages every 10s (but I guess that's okay since quick searches wont have large performance impact)
  2. if you get sid like this: admin__admin__search__RMD5624e8baf335d3eaf_1462017607.924, you'll need to shorten your search id.

HTH,
Bill

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...