topic Re: Hourly CPU spike on indexers in Getting Data In

Hourly CPU spike on indexers

Kindred — Fri, 21 Mar 2014 02:41:28 GMT

Hey,

Is there some internal scheduled event on an indexer than runs every hour? We're seeing our average CPU go from 30-40% to 100% during about a 5-10 minute period every hour on the hour.

splunkd.log doesn't reveal anything.

Re: Hourly CPU spike on indexers

linu1988 — Fri, 21 Mar 2014 05:24:34 GMT

if you are having hourly report searches or hourly monitoring of any large files then you will see the spike. No big deal

Re: Hourly CPU spike on indexers

MuS — Fri, 21 Mar 2014 07:18:07 GMT

Hi Kindred,

install the S.o.S app it can help you finding the cause for those peaks.

cheers, MuS

Re: Hourly CPU spike on indexers

Kindred — Sat, 22 Mar 2014 03:25:41 GMT

Not having much luck with SoS, the CPU report is all blank for some reason. Not sure what else it can provide.

Re: Hourly CPU spike on indexers

bmacias84 — Mon, 28 Sep 2020 16:12:35 GMT

Are you sure its a Splunk process? If you are running nix server I would monitor all process with the Nix_TA or if a windows system windows_TA. Set the collection interval to 1min.

Re: Hourly CPU spike on indexers

Kindred — Sat, 22 Mar 2014 11:13:22 GMT

@bmacias84 - yes its definitely Splunk, I can see it consume cpu by watching 'top'

Re: Hourly CPU spike on indexers

MuS — Sat, 22 Mar 2014 11:23:27 GMT

did you enable the cpu.sh input?

Re: Hourly CPU spike on indexers

Kindred — Tue, 25 Mar 2014 04:26:45 GMT

Doh.. thanks @MuS, enabled it and can see at least it is searches that are causing the CPU spike, but I can't drill down to find out which search. The only searches it seems to list are those local to the indexer, not the distributed searches from the search head.

Re: Hourly CPU spike on indexers

MuS — Tue, 25 Mar 2014 07:35:05 GMT

S.o.S is usually installed on every Splunk install, so to check the SH best thing to do is install S.o.S on them as well

Re: Hourly CPU spike on indexers

andykuhn — Fri, 22 Apr 2016 14:19:19 GMT

You may want to interrogate the splunk indexer's contributions to the _internal index as a timechart by SOURCE. The difference in log events by time should correspond to to your hourly CPU temper tantrum. Hopefully you can see a periodic difference in the number of events by source, which may help you identify events that only occur in this span.

Do you have any batch operations indexing data every hour....maybe being directed to only one indexer instead of being load-balanced?