https://community.splunk.com/t5/Getting-Data-In/Need-assistance-getting-large-data-into-Splunk/m-p/509718#M86693 <P>This is a note my client sent:</P><P>I can’t really use the logs from Splunk, because they are spotty.. Think I figured out why it’s missing data though.&nbsp; From our last meeting, they bumped up the transfer rate from 1024KBps to 2048KBps because the CE was generating too much data.&nbsp; However, the CE raw logs generate roughly about 75MB/Minute or 1.25MBps raw.&nbsp; Since Splunk does use compression at ~17%, the 2KBps gets you about 11KBps raw.&nbsp; The 1.25MBps is roughly 217.6KBps compressed.&nbsp; Hence, it gives you roughly 5% of the total logs.&nbsp; To support Openway and prevent dropping any logs, we need to set a conservative rate to handle 110MB/minute raw, which would be ~319KBps.</P><P>We did set in the limits.conf to the 2048 mentioned above, but we're not certain how to set to a level the client needs.</P> Fri, 17 Jul 2020 14:18:52 GMT nls7010 2020-07-17T14:18:52Z https://community.splunk.com/t5/Getting-Data-In/Need-assistance-getting-large-data-into-Splunk/m-p/509718#M86693 <P>This is a note my client sent:</P><P>I can’t really use the logs from Splunk, because they are spotty.. Think I figured out why it’s missing data though.&nbsp; From our last meeting, they bumped up the transfer rate from 1024KBps to 2048KBps because the CE was generating too much data.&nbsp; However, the CE raw logs generate roughly about 75MB/Minute or 1.25MBps raw.&nbsp; Since Splunk does use compression at ~17%, the 2KBps gets you about 11KBps raw.&nbsp; The 1.25MBps is roughly 217.6KBps compressed.&nbsp; Hence, it gives you roughly 5% of the total logs.&nbsp; To support Openway and prevent dropping any logs, we need to set a conservative rate to handle 110MB/minute raw, which would be ~319KBps.</P><P>We did set in the limits.conf to the 2048 mentioned above, but we're not certain how to set to a level the client needs.</P> Fri, 17 Jul 2020 14:18:52 GMT https://community.splunk.com/t5/Getting-Data-In/Need-assistance-getting-large-data-into-Splunk/m-p/509718#M86693 nls7010 2020-07-17T14:18:52Z