Getting Data In
Highlighted

Does Splunk process events before sending to nullqueue?

Path Finder

We have configured a default null queue to discard all events that we don't want to allow to be indexed without authorization. Our transforms have the first filter in transform to send to the nullqeuefilter.

The topic for this question is because I have events in our splunkd.log from a source which looks to be exceeding the linecount:

2/23/16 
5:34:11.169 PM  
02-23-2016 17:34:11.169 +0100 WARN  AggregatorMiningProcessor - Breaking event because limit of 256 has been exceeded 

But when I look at this specific sourcetype and data source, there are no events with more than one line, so my assumption is that those events exceeding the limits are those which are being discarded, in other words sent to the null queue. Is this the correct behavior ?

0 Karma
Highlighted

Re: Does Splunk process events before sending to nullqueue?

Splunk Employee
Splunk Employee

yes, if you look at the pipeline order , the nullqueue is happening after the parsing.
But before the license count.

Parsing (linebreak, encoding) -> aggregating (timestamp detection, event grouping )-> typing (regex, nullqueues, sedcmd ...) -> indexing (write to disk, count on license or forwarder on heavy forwarders)

View solution in original post

Highlighted

Re: Does Splunk process events before sending to nullqueue?

Path Finder

is not Splunk in this case wasting resources parsing events that will be sent to trash ?

0 Karma
Highlighted

Re: Does Splunk process events before sending to nullqueue?

Splunk Employee
Splunk Employee

correct. the best method is not avoid monitoring the bad event in the first place.

But we cannot filter an event before it has been break down to a proper event, you do not want to delete a whole file if only one event match the filter 🙂

0 Karma