https://community.splunk.com/t5/Getting-Data-In/How-much-data-can-i-index-per-second-on-a-single-indexer/m-p/13889#M1307 <P>Hey Gerald, i know this is a really old question but did you mean target indexed value of <BR /> 3 - 6 "kbps" or target "mbps" ?</P> Thu, 19 Jan 2012 22:02:21 GMT sonicZ 2012-01-19T22:02:21Z https://community.splunk.com/t5/Getting-Data-In/How-much-data-can-i-index-per-second-on-a-single-indexer/m-p/13886#M1304 <P>I've already got my single indexer spec'd to handle under 100Gigs a day and it meets the requirements. However i am getting blocked queue's at certain times of the day. What gives?</P> Thu, 20 May 2010 02:25:34 GMT https://community.splunk.com/t5/Getting-Data-In/How-much-data-can-i-index-per-second-on-a-single-indexer/m-p/13886#M1304 Chris_R_ 2010-05-20T02:25:34Z https://community.splunk.com/t5/Getting-Data-In/How-much-data-can-i-index-per-second-on-a-single-indexer/m-p/13887#M1305 <P>Splunk recommends indexing anywhere from 3-10mb per second on a single indexer. Please keep in mind the upper limit of 10mbps is on very fast hardware, 15k rpm disks, raid 0+1 array, fast bonnie++ results </P> <P>Your system may be indexing within the reccomendations of &lt; 100gig per day spec'd box, but if you have blocked indexqueue's at certain times you may be indexing in too much data at certain time frames. </P> <P>Check your queues with this search during problem time frames.<BR /> index=_internal source="*metrics.log*" group=queue | timechart perc95(current_size) by name</P> <P>If you want to drill down and find out the maximum kbps indexed at that time<BR /> index="_internal" source="*metrics.log*" per_index_thruput | timechart span=1h max(kbps) by series | addtotals </P> <P>You can then identify heavy forwarders sending lots of data<BR /> index=_internal source="*metrics.log*" per_host_thruput | eval mb=(kb/1024) | timechart span=1h sum(mb) by series | addtotals</P> <P>For further assistance and recommendations on how to increase performance open a case with support.</P> Thu, 20 May 2010 02:27:13 GMT https://community.splunk.com/t5/Getting-Data-In/How-much-data-can-i-index-per-second-on-a-single-indexer/m-p/13887#M1305 Chris_R_ 2010-05-20T02:27:13Z https://community.splunk.com/t5/Getting-Data-In/How-much-data-can-i-index-per-second-on-a-single-indexer/m-p/13888#M1306 <P>Most of the time, if you are not reaching the target kbps indexed (i.e., three to six kb per second - 10 is possible, but not easy to achieve), it's either because of your disk performance, or because you have poor index-time rules. To achieve the best index thruput, you should optimize:</P> <UL> <LI>Timestamp extraction: use explicit timestamp prefixes, formats, and lookaheads as much as possible</LI> <LI>Line breaking rules: try to use LINE_BREAKER and avoid LINE_MERGING if possible, and keep the merging rules simple if not</LI> <LI>Index-time transforms: Have as few as and simple index-time transforms (for sources, hosts, index, or other fields) as possible</LI> <LI>Regular expressions: Make sure your regular expressions are PCRE-efficient </LI> </UL> Thu, 20 May 2010 18:41:29 GMT https://community.splunk.com/t5/Getting-Data-In/How-much-data-can-i-index-per-second-on-a-single-indexer/m-p/13888#M1306 gkanapathy 2010-05-20T18:41:29Z https://community.splunk.com/t5/Getting-Data-In/How-much-data-can-i-index-per-second-on-a-single-indexer/m-p/13889#M1307 <P>Hey Gerald, i know this is a really old question but did you mean target indexed value of <BR /> 3 - 6 "kbps" or target "mbps" ?</P> Thu, 19 Jan 2012 22:02:21 GMT https://community.splunk.com/t5/Getting-Data-In/How-much-data-can-i-index-per-second-on-a-single-indexer/m-p/13889#M1307 sonicZ 2012-01-19T22:02:21Z