https://community.splunk.com/t5/Getting-Data-In/How-to-send-lines-to-the-nullQueue-before-applying-line-breaking/m-p/130116#M26766 <P>Assuming that <A href="http://wiki.splunk.com/Community:HowIndexingWorks">the diagram on this page</A> is still up to date, then the nullqueue comes into effect after parsing and aggregation queue during typing queue. I don't think it's possible to change this within splunk.</P> Thu, 16 Jul 2015 10:34:09 GMT jeffland 2015-07-16T10:34:09Z https://community.splunk.com/t5/Getting-Data-In/How-to-send-lines-to-the-nullQueue-before-applying-line-breaking/m-p/130114#M26764 <P>An app of ours spits such a huge volume of data when our Devs increase its debug level to Trace that it essentially results on the Splunk Heavy forwarders sitting on those servers to stall and becoming non operational<BR /> (we implemented a script to auto restart them when we don't have indexed data on those servers for more than 10m)</P> <P>As far as I know it's only possible to send <STRONG>events</STRONG> to nullQueuing, ie, after properties like line breaking (which is costly but vital for us for those logs) have already been computed.</P> <P>Any way we can setup props / transforms / anything to first completely ignore the TRACE lines and only then applying the props to the logs?</P> Wed, 04 Feb 2015 15:30:37 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-send-lines-to-the-nullQueue-before-applying-line-breaking/m-p/130114#M26764 splunk_zen 2015-02-04T15:30:37Z https://community.splunk.com/t5/Getting-Data-In/How-to-send-lines-to-the-nullQueue-before-applying-line-breaking/m-p/130115#M26765 <P>Hi,</P> <P>You could setup a props and transforms to channel such events to null queue</P> <P>props.conf<BR /> [source::/.../debug]<BR /> TRANSFORMS-filterdata=set_nullqueue_debug</P> <P>transforms.conf<BR /> [set_nullqueue_debug]<BR /> REGEX=(debug)<BR /> DEST_KEY=queue<BR /> FORMAT=nullQueue</P> <P>You would need to create a regex which matches all the debug events from source debug and channel them to nullqueue. All other events will get indexed.</P> Tue, 29 Sep 2020 06:42:57 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-send-lines-to-the-nullQueue-before-applying-line-breaking/m-p/130115#M26765 merp96 2020-09-29T06:42:57Z https://community.splunk.com/t5/Getting-Data-In/How-to-send-lines-to-the-nullQueue-before-applying-line-breaking/m-p/130116#M26766 <P>Assuming that <A href="http://wiki.splunk.com/Community:HowIndexingWorks">the diagram on this page</A> is still up to date, then the nullqueue comes into effect after parsing and aggregation queue during typing queue. I don't think it's possible to change this within splunk.</P> Thu, 16 Jul 2015 10:34:09 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-send-lines-to-the-nullQueue-before-applying-line-breaking/m-p/130116#M26766 jeffland 2015-07-16T10:34:09Z https://community.splunk.com/t5/Getting-Data-In/How-to-send-lines-to-the-nullQueue-before-applying-line-breaking/m-p/130117#M26767 <P>No, you can't use nullqueue before linebreaking - because only after your data is broken into lines can you throw lines away/into the nullqueue. Before that, you could of course apply a regex to determine where to route stuff, but you would then apply those settings to various lines and events (for example if you're reading a file, those are read in chunks of 64k - would you like to apply the routing to that chunk?)<BR /> If you desire deeper insight, I would recommend the talk "How splunkd works" from .conf2014, check it out <A href="http://conf.splunk.com/speakers/2014.html">here</A>. It details how the different queues are applied in order and how they work together pretty well.</P> Thu, 16 Jul 2015 13:05:29 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-send-lines-to-the-nullQueue-before-applying-line-breaking/m-p/130117#M26767 jeffland 2015-07-16T13:05:29Z https://community.splunk.com/t5/Getting-Data-In/How-to-send-lines-to-the-nullQueue-before-applying-line-breaking/m-p/130118#M26768 <P>Yeah, was aware of the Splunk data flow, was just curious if someone had implemented something which circumvented this. Thank you anyway Jeff</P> Fri, 17 Jul 2015 10:41:01 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-send-lines-to-the-nullQueue-before-applying-line-breaking/m-p/130118#M26768 splunk_zen 2015-07-17T10:41:01Z https://community.splunk.com/t5/Getting-Data-In/How-to-send-lines-to-the-nullQueue-before-applying-line-breaking/m-p/130119#M26769 <P>Thanks but that was our first approach, which didn't answer my requirement of dropping those events before the line breaking. The reason why this was so important was this app function logging was poorly written and was generating <EM>one line</EM> with millions of characters</P> Fri, 17 Jul 2015 10:44:11 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-send-lines-to-the-nullQueue-before-applying-line-breaking/m-p/130119#M26769 splunk_zen 2015-07-17T10:44:11Z https://community.splunk.com/t5/Getting-Data-In/How-to-send-lines-to-the-nullQueue-before-applying-line-breaking/m-p/130120#M26770 <P>Thanks Jeff, that was my initial assumption.</P> Fri, 17 Jul 2015 10:44:33 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-send-lines-to-the-nullQueue-before-applying-line-breaking/m-p/130120#M26770 splunk_zen 2015-07-17T10:44:33Z