Join the Conversation
- Find Answers
- :
- Splunk Administration
- :
- Monitoring Splunk
- :
- Re: splunkd is at 100% cpu, lag in indexing
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
Yesterday I noticed that there's a substantial lag in event indexing to the default index (main). When inspecting the processes on my machine, I noticed that a single splunkd process running at 100% cpu.
I tried to restart splunk several time, but splunk gets to 100% cpu immediately and stays there.
I've opened ticket at support@splunk.com, but no response till now.
After deleting the index and re-indexing things got back to normal.
24h later I'm encountering the same problem.
The machine splunk is running on has 16 CPUs, 8GB RAM. ~200MB of data is being indexed daily, and there's no significant search activity as well.
Will appreciate any help with that.
Oren.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Issue solved.
The problem was that Splunk was monitoring more than 4000 log files which were already indexed.
In inputs.conf, we simply added ignoreOlderThan = 2d which solved the problem.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Inputs .conf file exists in the host and not in the indexer and this is for the specific host only TA
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Issue solved.
The problem was that Splunk was monitoring more than 4000 log files which were already indexed.
In inputs.conf, we simply added ignoreOlderThan = 2d which solved the problem.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
I assume this is not a Splunk indexer. right ?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
the inputs.conf file exists in the host and this change need to be made at the host level. this will not affect the indexing
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks for the solution.
Changes made....Problem went away.
Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!
Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.
[Puzzles] Solve, Learn, Repeat: Character substitutions with Regular Expressions
Splunk Community Badges!
[Puzzles] Solve, Learn, Repeat: Matching cron expressions
