I am investigating higher CPU usage on my indexers, and am finding that this is a hard topic to really pinpoint.
I run this search on my search head to identify different searches and the resource ...
See more...
I am investigating higher CPU usage on my indexers, and am finding that this is a hard topic to really pinpoint.
I run this search on my search head to identify different searches and the resource consumption, but the results are confusing me.
index=_introspection host=* source=*/resource_usage.log* component=PerProcess data.process_type="search"
| stats latest(data.pct_cpu) AS resource_usage_cpu latest(data.mem_used) AS resource_usage_mem by _time, data.search_props.type,data.search_props.mode,data.search_props.user, data.search_props.app, host data.search_props.label data.elapsed data.search_props.search_head
| sort - resource_usage_cpu
_time
data.search_props.type
data.search_props.mode
host
data.search_props.label
data.elapsed
data.search_props.search_head
resource_usage_cpu
2022-11-01 10:23:54.338
scheduled
historical batch
idx04-k
Process-Creation-Events-DomainController
1431.6000
sh02-g
95.40
2022-11-01 10:23:52.815
scheduled
historical batch
idx03-k
Process-Creation-Events-DomainController
1430.0200
sh02-g
115.50
2022-11-01 10:23:50.738
scheduled
historical batch
idx05-k
Process-Creation-Events-DomainController
1427.9800
sh02-g
105.70
2022-11-01 10:23:46.748
scheduled
historical batch
idx03-g
Process-Creation-Events-DomainController
1424.0400
sh02-g
101.90
2022-11-01 10:23:45.081
scheduled
historical batch
idx02-k
Process-Creation-Events-DomainController
1422.3200
sh02-g
97.90
From this, I can see that the search:
1) Was triggered from sh02
2) Was executed across several my indexers
3) Took ~1500 seconds to run
4) Consumed ~1 core on each instance
BUT:
The search is scheduled for once a day, and that time is not 10:23. It is scheduled for 11. (No window)
There are dozens on "instances" of this search being executed on all 10 of my indexers, triggered by sh02, in the ~10:22 timeframe. Maybe one row in the table above per indexer might make sense, but this is so many.
What is happening here? How do I read these results to make a sane performance judgement about this situation?