Getting Data In

Does Splunk process events before sending to nullqueue?

dsmc_adv
Path Finder

We have configured a default null queue to discard all events that we don't want to allow to be indexed without authorization. Our transforms have the first filter in transform to send to the null_qeue_filter.

The topic for this question is because I have events in our splunkd.log from a source which looks to be exceeding the linecount:

2/23/16 
5:34:11.169 PM  
02-23-2016 17:34:11.169 +0100 WARN  AggregatorMiningProcessor - Breaking event because limit of 256 has been exceeded 

But when I look at this specific sourcetype and data source, there are no events with more than one line, so my assumption is that those events exceeding the limits are those which are being discarded, in other words sent to the null queue. Is this the correct behavior ?

0 Karma
1 Solution

yannK
Splunk Employee
Splunk Employee

yes, if you look at the pipeline order , the nullqueue is happening after the parsing.
But before the license count.

Parsing (linebreak, encoding) -> aggregating (timestamp detection, event grouping )-> typing (regex, nullqueues, sedcmd ...) -> indexing (write to disk, count on license or forwarder on heavy forwarders)

View solution in original post

yannK
Splunk Employee
Splunk Employee

yes, if you look at the pipeline order , the nullqueue is happening after the parsing.
But before the license count.

Parsing (linebreak, encoding) -> aggregating (timestamp detection, event grouping )-> typing (regex, nullqueues, sedcmd ...) -> indexing (write to disk, count on license or forwarder on heavy forwarders)

dsmc_adv
Path Finder

is not Splunk in this case wasting resources parsing events that will be sent to trash ?

0 Karma

yannK
Splunk Employee
Splunk Employee

correct. the best method is not avoid monitoring the bad event in the first place.

But we cannot filter an event before it has been break down to a proper event, you do not want to delete a whole file if only one event match the filter 🙂

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...