Join the Conversation
- Find Answers
- :
- Splunk Platform
- :
- Splunk Enterprise
- :
- Re: how to quickly index historical logs on a dedi...
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I need to index several hundred gigs of historical logs. i have a machine that is dedicated for this purpose. i installed the universal forwarder and have used the [monitor] stanza in inputs.conf to start the indexing. it is working, but it seems REALLY slow. since this server is dedicated to this purpose is there any way i can force the forwarder to use more system resources to chug through the logs at a faster pace?
I'm also open to alternative solutions to this problem.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
There could be a few reasons as to the speed, disk IO on the forwarder, cpu, etc.
However the first thing I would look at is the limits.conf file. The universalForwarder has limits for how much data it can send at a time, this may be the cause of the perceived slowness.
http://www.splunk.com/base/Documentation/4.2.2/Admin/Limitsconf
[thruput]
maxKBps =
* If specified and not zero, this limits the speed through the thruput processor to the specified
rate in kilobytes per second.
* To control the CPU load while indexing, use this to throttle the number of events this indexer
processes to the rate (in KBps) you specify.
On a universal forwarder, this is set to 256 by default.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
There could be a few reasons as to the speed, disk IO on the forwarder, cpu, etc.
However the first thing I would look at is the limits.conf file. The universalForwarder has limits for how much data it can send at a time, this may be the cause of the perceived slowness.
http://www.splunk.com/base/Documentation/4.2.2/Admin/Limitsconf
[thruput]
maxKBps =
* If specified and not zero, this limits the speed through the thruput processor to the specified
rate in kilobytes per second.
* To control the CPU load while indexing, use this to throttle the number of events this indexer
processes to the rate (in KBps) you specify.
On a universal forwarder, this is set to 256 by default.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
you guys are both right- it was my thruput. as soon as i bumped it up i could process logs way faster
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
damn IE not having my credentials cached...
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Did you raise the maxKbps setting in the [thruput] stanza of limits.conf?
http://www.splunk.com/base/Documentation/latest/Admin/Limitsconf
Upcoming Webinar: Unmasking Insider Threats with Slunk Enterprise Security’s UEBA
.conf25 technical session recap of Observability for Gen AI: Monitoring LLM ...
A Season of Skills: New Splunk Courses to Light Up Your Learning Journey
