All Apps and Add-ons

WARN TailReader - Could not send data to output queue (parsingQueue), retrying...

meghasinghal
Engager

We have filtered out a large amount of firewall logs on heavy forwarder due to which we are receiving warning "WARN TailReader - Could not send data to output queue (parsingQueue), retrying...". Could anyone please help if we want to whitelist the data instead of blacklisting, so that this issue co

Labels (1)
0 Karma

harsmarvania57
Ultra Champion

Hi,

You need to provide more information, what you have blacklisted and how you have blacklisted ? What type of resources you have on Heavy Forwarder and Indexer ? How much data firewall is generating ?

0 Karma

meghasinghal
Engager

We have blacklisted some DNS consuming around 30 GB of license and the actions such as timeout, accept, close consuming 70 GB of license. Before the license consumption by firewall logs was 130 GB and is now reduced to 30 GB. We are only taking two action into consideration.

[transforms]
REGEX = 
DEST_KEY = queue
FORMAT = nullQueue

We have filtered out the data by sending it into null queue. We have 8 Core CPU on H.F, 16 GB RAM.

0 Karma

harsmarvania57
Ultra Champion

Can you please search blocked=true in $SPLUNK_HOME/var/log/splunk/metrics.log on HF ? Also can you please provide REGEX which you are using (mask any sensitive data).

 

0 Karma

meghasinghal
Engager

I checked for blocked=true

Metrics - group=queue, name=udp_queue, blocked=true, max_size_kb=500, current_size_kb=499, current_size=554, largest_size=588, smallest_size=0
Metrics - group=queue, name=parsingqueue, blocked=true, max_size_kb=25600, current_size_kb=25599, current_size=28678, largest_size=28678, smallest_size=23788
Metrics - group=queue, name=aggqueue, blocked=true, max_size_kb=25600, current_size_kb=25599, current_size=32915, largest_size=35086, smallest_size=28916
Metrics - group=queue, name=splunktcpin, blocked=true, max_size_kb=500, current_size_kb=499, current_size=710, largest_size=751, smallest_size=0

 

Regex=(?m).*(server).*device(p|q|r|s|t|u|v|w|x|.......).*vdom.*(a|b|c|d|e|f|g|h|.......).*type.*(traffic).*action.*(accept|client-rst|close|dns|ip-conn|server-rst|timeout).*

0 Karma

harsmarvania57
Ultra Champion

So it look like due to parsingqueue and aggregationqueue block, it back pressure udp and splunk_tcpin queue. I would like to suggest that please configure LINE_BREAKER, SHOULD_LINEMERGE, TIME_FORMAT TIME_PREFIX and MAX_TIMESTAMP_LOOKAHEAD parameter for all sources which are ingesting massive amount of data on HF.

In addition I'll suggest if you can write REGEX with least steps matching then REGEX engine will perform those REGEX faster.

Get Updates on the Splunk Community!

Leveraging Automated Threat Analysis Across the Splunk Ecosystem

Enhance Security Operations with Automated Threat Analysis in the Splunk EcosystemAre you leveraging ...

Splunk Developers: Go Beyond the Dashboard with These .Conf25 Sessions

  Whether you’re building custom apps, diving into SPL2, or integrating AI and machine learning into your ...

Index This | How do you write 23 only using the number 2?

July 2025 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this month’s ...