Monitoring Splunk

splunkd is at 100% cpu, lag in indexing

oreni
Explorer

Hello,

Yesterday I noticed that there's a substantial lag in event indexing to the default index (main). When inspecting the processes on my machine, I noticed that a single splunkd process running at 100% cpu.

I tried to restart splunk several time, but splunk gets to 100% cpu immediately and stays there.
I've opened ticket at support@splunk.com, but no response till now.
After deleting the index and re-indexing things got back to normal.

24h later I'm encountering the same problem.

The machine splunk is running on has 16 CPUs, 8GB RAM. ~200MB of data is being indexed daily, and there's no significant search activity as well.

Will appreciate any help with that.

Oren.

Tags (2)
1 Solution

oreni
Explorer

Issue solved.

The problem was that Splunk was monitoring more than 4000 log files which were already indexed.

In inputs.conf, we simply added ignoreOlderThan = 2d which solved the problem.

View solution in original post

arunsundaram
New Member

Inputs .conf file exists in the host and not in the indexer and this is for the specific host only TA

0 Karma

oreni
Explorer

Issue solved.

The problem was that Splunk was monitoring more than 4000 log files which were already indexed.

In inputs.conf, we simply added ignoreOlderThan = 2d which solved the problem.

View solution in original post

ali_alnajjar_ve
Explorer

Hi,

I assume this is not a Splunk indexer. right ?

0 Karma

arunsundaram
New Member

the inputs.conf file exists in the host and this change need to be made at the host level. this will not affect the indexing

0 Karma

campbells
Engager

Thanks for the solution.

Changes made....Problem went away.

0 Karma