Hi all, in splunk there is always this icon next to your user for the "Health of Splunk Deployment". You can change these indicators and futures or their teshholds, but I can't find anything about what splunk actually does in the background to collect these values. You can find something like this in health.conf: [feature:iowait] display_name = IOWait indicator:avg_cpu__max_perc_last_3m:description = This indicator tracks the average IOWait percentage across all CPUs on the machine running the Splunk Enterprise instance, over the last 3 minute window. By default, this indicator will turn Yellow if the percentage exceeds 1% and Red if it exceeds 3% during this window. indicator:avg_cpu__max_perc_last_3m:red = 3 indicator:avg_cpu__max_perc_last_3m:yellow = 1 indicator:single_cpu__max_perc_last_3m:description = This indicator tracks the IOWait percentage for the single most bottle-necked CPU on the machine running the Splunk Enterprise instance, over the last 3 minute window. By default, this indicator will turn Yellow if the percentage exceeds 5% and Red if it exceeds 10% during this window. indicator:single_cpu__max_perc_last_3m:red = 10 indicator:single_cpu__max_perc_last_3m:yellow = 5 indicator:sum_top3_cpu_percs__max_last_3m:description = This indicator tracks the sum of IOWait percentage for the three most bottle-necked CPUs on the machine running the Splunk Enterprise instance, over the last 3 minute window. By default, this indicator will turn Yellow if the sum exceeds 7% and Red if it exceeds 15% during this window. indicator:sum_top3_cpu_percs__max_last_3m:red = 15 indicator:sum_top3_cpu_percs__max_last_3m:yellow = 7 I can´t find out how splunk generate this data and how this alert or indicator is created. There must be a kind of process like scripted input which execute a top command to look for the cpu wait time write it to the health.log in splunk ingest this log and a search which provide the information for these indicator.
... View more