Archive

how to quickly index historical logs on a dedicated forwarder

Communicator

I need to index several hundred gigs of historical logs. i have a machine that is dedicated for this purpose. i installed the universal forwarder and have used the [monitor] stanza in inputs.conf to start the indexing. it is working, but it seems REALLY slow. since this server is dedicated to this purpose is there any way i can force the forwarder to use more system resources to chug through the logs at a faster pace?

I'm also open to alternative solutions to this problem.

0 Karma
1 Solution

Motivator

There could be a few reasons as to the speed, disk IO on the forwarder, cpu, etc.

However the first thing I would look at is the limits.conf file. The universalForwarder has limits for how much data it can send at a time, this may be the cause of the perceived slowness.

http://www.splunk.com/base/Documentation/4.2.2/Admin/Limitsconf

[thruput]

maxKBps =
* If specified and not zero, this limits the speed through the thruput processor to the specified
rate in kilobytes per second.
* To control the CPU load while indexing, use this to throttle the number of events this indexer
processes to the rate (in KBps) you specify.

On a universal forwarder, this is set to 256 by default.

View solution in original post

Motivator

There could be a few reasons as to the speed, disk IO on the forwarder, cpu, etc.

However the first thing I would look at is the limits.conf file. The universalForwarder has limits for how much data it can send at a time, this may be the cause of the perceived slowness.

http://www.splunk.com/base/Documentation/4.2.2/Admin/Limitsconf

[thruput]

maxKBps =
* If specified and not zero, this limits the speed through the thruput processor to the specified
rate in kilobytes per second.
* To control the CPU load while indexing, use this to throttle the number of events this indexer
processes to the rate (in KBps) you specify.

On a universal forwarder, this is set to 256 by default.

View solution in original post

Communicator

you guys are both right- it was my thruput. as soon as i bumped it up i could process logs way faster

0 Karma

Motivator

damn IE not having my credentials cached...

SplunkTrust
SplunkTrust

Did you raise the maxKbps setting in the [thruput] stanza of limits.conf?

http://www.splunk.com/base/Documentation/latest/Admin/Limitsconf

State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!