Getting Data In

How to send lines to the nullQueue *before* applying line breaking

Builder

An app of ours spits such a huge volume of data when our Devs increase its debug level to Trace that it essentially results on the Splunk Heavy forwarders sitting on those servers to stall and becoming non operational
(we implemented a script to auto restart them when we don't have indexed data on those servers for more than 10m)

As far as I know it's only possible to send events to nullQueuing, ie, after properties like line breaking (which is costly but vital for us for those logs) have already been computed.

Any way we can setup props / transforms / anything to first completely ignore the TRACE lines and only then applying the props to the logs?

0 Karma
1 Solution

Champion

No, you can't use nullqueue before linebreaking - because only after your data is broken into lines can you throw lines away/into the nullqueue. Before that, you could of course apply a regex to determine where to route stuff, but you would then apply those settings to various lines and events (for example if you're reading a file, those are read in chunks of 64k - would you like to apply the routing to that chunk?)
If you desire deeper insight, I would recommend the talk "How splunkd works" from .conf2014, check it out here. It details how the different queues are applied in order and how they work together pretty well.

View solution in original post

Champion

No, you can't use nullqueue before linebreaking - because only after your data is broken into lines can you throw lines away/into the nullqueue. Before that, you could of course apply a regex to determine where to route stuff, but you would then apply those settings to various lines and events (for example if you're reading a file, those are read in chunks of 64k - would you like to apply the routing to that chunk?)
If you desire deeper insight, I would recommend the talk "How splunkd works" from .conf2014, check it out here. It details how the different queues are applied in order and how they work together pretty well.

View solution in original post

Builder

Thanks Jeff, that was my initial assumption.

0 Karma

Champion

Assuming that the diagram on this page is still up to date, then the nullqueue comes into effect after parsing and aggregation queue during typing queue. I don't think it's possible to change this within splunk.

Builder

Yeah, was aware of the Splunk data flow, was just curious if someone had implemented something which circumvented this. Thank you anyway Jeff

0 Karma

Path Finder

Hi,

You could setup a props and transforms to channel such events to null queue

props.conf
[source::/.../debug]
TRANSFORMS-filterdata=setnullqueuedebug

transforms.conf
[setnullqueuedebug]
REGEX=(debug)
DEST_KEY=queue
FORMAT=nullQueue

You would need to create a regex which matches all the debug events from source debug and channel them to nullqueue. All other events will get indexed.

0 Karma

Builder

Thanks but that was our first approach, which didn't answer my requirement of dropping those events before the line breaking. The reason why this was so important was this app function logging was poorly written and was generating one line with millions of characters

0 Karma