how to quickly index historical logs on a dedicated forwarder

[tpsplunk]

I need to index several hundred gigs of historical logs. i have a machine that is dedicated for this purpose. i installed the universal forwarder and have used the [monitor] stanza in inputs.conf to start the indexing. it is working, but it seems REALLY slow. since this server is dedicated to this purpose is there any way i can force the forwarder to use more system resources to chug through the logs at a faster pace?

I'm also open to alternative solutions to this problem.

[mikelanghorst · Accepted solution]

There could be a few reasons as to the speed, disk IO on the forwarder, cpu, etc.

However the first thing I would look at is the limits.conf file. The universalForwarder has limits for how much data it can send at a time, this may be the cause of the perceived slowness.

http://www.splunk.com/base/Documentation/4.2.2/Admin/Limitsconf

[thruput]

maxKBps =
* If specified and not zero, this limits the speed through the thruput processor to the specified
rate in kilobytes per second.
* To control the CPU load while indexing, use this to throttle the number of events this indexer
processes to the rate (in KBps) you specify.

On a universal forwarder, this is set to 256 by default.

[tpsplunk]

you guys are both right- it was my thruput. as soon as i bumped it up i could process logs way faster

[mikelanghorst]

damn IE not having my credentials cached...

[dwaddle]

Did you raise the maxKbps setting in the [thruput] stanza of limits.conf?

http://www.splunk.com/base/Documentation/latest/Admin/Limitsconf