Hourly CPU spike on indexers

[Kindred]

Hey,

Is there some internal scheduled event on an indexer than runs every hour? We're seeing our average CPU go from 30-40% to 100% during about a 5-10 minute period every hour on the hour.

splunkd.log doesn't reveal anything.

[andykuhn]

You may want to interrogate the splunk indexer's contributions to the _internal index as a timechart by SOURCE. The difference in log events by time should correspond to to your hourly CPU temper tantrum. Hopefully you can see a periodic difference in the number of events by source, which may help you identify events that only occur in this span.

Do you have any batch operations indexing data every hour....maybe being directed to only one indexer instead of being load-balanced?

[Kindred]

@bmacias84 - yes its definitely Splunk, I can see it consume cpu by watching 'top'

[bmacias84]

Are you sure its a Splunk process? If you are running nix server I would monitor all process with the Nix_TA or if a windows system windows_TA. Set the collection interval to 1min.

[MuS]

Hi Kindred,

install the S.o.S app it can help you finding the cause for those peaks.

cheers, MuS

[MuS]

S.o.S is usually installed on every Splunk install, so to check the SH best thing to do is install S.o.S on them as well

[Kindred]

Doh.. thanks @MuS, enabled it and can see at least it is searches that are causing the CPU spike, but I can't drill down to find out which search. The only searches it seems to list are those local to the indexer, not the distributed searches from the search head.

[MuS]

did you enable the cpu.sh input?

[Kindred]

Not having much luck with SoS, the CPU report is all blank for some reason. Not sure what else it can provide.

[linu1988]

if you are having hourly report searches or hourly monitoring of any large files then you will see the spike. No big deal