Hi guys, I am currently seeing that processing queues on one of my heavy forwarders appear to be blocking during a 6 hour period at night time when log volume being ingested is much lower (during this period, log volume ingested drops from 10 million to under 3 million events). Are there are obvious reasons as to why queue block ratios would increase at same time that thruput/event volume decreases? As I'm guessing that the opposite would generally be expected? We can see that block ratios increase at 1:00 AM below: index=_internal source=*metrics.log group=queue blocked=true host=HF max_size_kb>0 | timechart span=30m@m count by name At the same time we can see thruput decrease below: index=_internal sourcetype=splunkd host=HF group=per_sourcetype_thruput series=* | timechart sum(kb) by series Current queue config and context over last 24h: I have also noticed a lot of 'Could not send data to output queue (parsingQueue), retrying' errors around the same time: index=_internal host=HFsource=*splunkd* log_level=ERROR OR log_level=WARN event_message="Could not send data to output queue (parsingQueue), retrying..." | timechart span=5m@m count by event_message I would appreciate any answers around why queue block ratios would increase at same time that thruput/event volume decreases and also any solutions for getting average queue block ratios as close to 0 as possible - queues currently appear to be blocking throughout the day with highest block ratios occuring 1:00 - 8:00 AM.
... View more
