I need to index several hundred gigs of historical logs. i have a machine that is dedicated for this purpose. i installed the universal forwarder and have used the [monitor] stanza in inputs.conf to start the indexing. it is working, but it seems REALLY slow. since this server is dedicated to this purpose is there any way i can force the forwarder to use more system resources to chug through the logs at a faster pace?
I'm also open to alternative solutions to this problem.
There could be a few reasons as to the speed, disk IO on the forwarder, cpu, etc.
However the first thing I would look at is the limits.conf file. The universalForwarder has limits for how much data it can send at a time, this may be the cause of the perceived slowness.
http://www.splunk.com/base/Documentation/4.2.2/Admin/Limitsconf
[thruput]
maxKBps =
* If specified and not zero, this limits the speed through the thruput processor to the specified
rate in kilobytes per second.
* To control the CPU load while indexing, use this to throttle the number of events this indexer
processes to the rate (in KBps) you specify.
On a universal forwarder, this is set to 256 by default.
There could be a few reasons as to the speed, disk IO on the forwarder, cpu, etc.
However the first thing I would look at is the limits.conf file. The universalForwarder has limits for how much data it can send at a time, this may be the cause of the perceived slowness.
http://www.splunk.com/base/Documentation/4.2.2/Admin/Limitsconf
[thruput]
maxKBps =
* If specified and not zero, this limits the speed through the thruput processor to the specified
rate in kilobytes per second.
* To control the CPU load while indexing, use this to throttle the number of events this indexer
processes to the rate (in KBps) you specify.
On a universal forwarder, this is set to 256 by default.
you guys are both right- it was my thruput. as soon as i bumped it up i could process logs way faster
damn IE not having my credentials cached...
Did you raise the maxKbps
setting in the [thruput]
stanza of limits.conf?
http://www.splunk.com/base/Documentation/latest/Admin/Limitsconf