We have configured a default null queue to discard all events that we don't want to allow to be indexed without authorization. Our transforms have the first filter in transform to send to the nullqeuefilter.
The topic for this question is because I have events in our splunkd.log from a source which looks to be exceeding the linecount:
2/23/16 5:34:11.169 PM 02-23-2016 17:34:11.169 +0100 WARN AggregatorMiningProcessor - Breaking event because limit of 256 has been exceeded
But when I look at this specific sourcetype and data source, there are no events with more than one line, so my assumption is that those events exceeding the limits are those which are being discarded, in other words sent to the null queue. Is this the correct behavior ?
yes, if you look at the pipeline order , the nullqueue is happening after the parsing.
But before the license count.
Parsing (linebreak, encoding) -> aggregating (timestamp detection, event grouping )-> typing (regex, nullqueues, sedcmd ...) -> indexing (write to disk, count on license or forwarder on heavy forwarders)
correct. the best method is not avoid monitoring the bad event in the first place.
But we cannot filter an event before it has been break down to a proper event, you do not want to delete a whole file if only one event match the filter 🙂