Monitoring Splunk

splunkd is at 100% cpu, lag in indexing

oreni
Explorer

Hello,

Yesterday I noticed that there's a substantial lag in event indexing to the default index (main). When inspecting the processes on my machine, I noticed that a single splunkd process running at 100% cpu.

I tried to restart splunk several time, but splunk gets to 100% cpu immediately and stays there.
I've opened ticket at support@splunk.com, but no response till now.
After deleting the index and re-indexing things got back to normal.

24h later I'm encountering the same problem.

The machine splunk is running on has 16 CPUs, 8GB RAM. ~200MB of data is being indexed daily, and there's no significant search activity as well.

Will appreciate any help with that.

Oren.

Tags (2)
1 Solution

oreni
Explorer

Issue solved.

The problem was that Splunk was monitoring more than 4000 log files which were already indexed.

In inputs.conf, we simply added ignoreOlderThan = 2d which solved the problem.

View solution in original post

arunsundaram
New Member

Inputs .conf file exists in the host and not in the indexer and this is for the specific host only TA

0 Karma

oreni
Explorer

Issue solved.

The problem was that Splunk was monitoring more than 4000 log files which were already indexed.

In inputs.conf, we simply added ignoreOlderThan = 2d which solved the problem.

ali_alnajjar_ve
Explorer

Hi,

I assume this is not a Splunk indexer. right ?

0 Karma

arunsundaram
New Member

the inputs.conf file exists in the host and this change need to be made at the host level. this will not affect the indexing

0 Karma

campbells
Engager

Thanks for the solution.

Changes made....Problem went away.

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...