topic Re: Events in a single file are not getting evenly distributed to multiple indexers in Getting Data In

Events in a single file are not getting evenly distributed to multiple indexers

ishugupta — Wed, 29 Apr 2015 17:19:18 GMT

I have a light weight forwarder pointing two indexers . I get a batch data everyday in a single file . The file size is 28 GB on an average .
All the events in that file gets ingested in one single indexer . I noticed splunk is not able to distribute the events within a single file . It distributes events to indexers based on file level .So in case I get 10- 11 file on some day , one single file gets ingested to one single indexer and other file on other indexer. Is there any solution that can help me distribute the data on event level.
I am using Splunk 6.1.7

Re: Events in a single file are not getting evenly distributed to multiple indexers

satishsdange — Thu, 30 Apr 2015 10:09:10 GMT

LWF is deprecated. Please replace it by UF.

Re: Events in a single file are not getting evenly distributed to multiple indexers

Runals — Thu, 30 Apr 2015 11:32:53 GMT

I suspect what you are running into is a forwarder will send data to an indexer for 30s OR until it hits an end of file and then switch. If these are single files as part of a batch job I'm not surprised you are seeing the behavior you are describing. I'm not sure there is a good way to adjust that. You could check out the forceTimebasedAutoLB setting but I'm not sure this is the best use case for that setting.

Re: Events in a single file are not getting evenly distributed to multiple indexers

ishugupta — Thu, 30 Apr 2015 21:58:20 GMT

thanks Runals , Seems like there is not concrete solution to this . I will have to break my files to multiple smaller files .
I do not want to change the forceTimebasedAutoLB setting , as it might truncate an event itself.