How do I verify if splunk forwarder dropping event from the queue? Splunkd log _internal.

[abi2023]

In my index I don't see all the logs being forwarder by the Splunk UF. How can monitor when event is drop from event queue on the Spunk UF.

Can I monitor this in Splunk Deployment server?

[gcusello · Accepted solution]

Hi @abi2023 ,

using the following search, you can see if you have full queues:

index=_internal  source=*metrics.log sourcetype=splunkd group=queue 
| eval name=case(name=="aggqueue","2 - Aggregation Queue",
 name=="indexqueue", "4 - Indexing Queue",
 name=="parsingqueue", "1 - Parsing Queue",
 name=="typingqueue", "3 - Typing Queue",
 name=="splunktcpin", "0 - TCP In Queue",
 name=="tcpin_cooked_pqueue", "0 - TCP In Queue") 
| eval max=if(isnotnull(max_size_kb),max_size_kb,max_size) 
| eval curr=if(isnotnull(current_size_kb),current_size_kb,current_size) 
| eval fill_perc=round((curr/max)*100,2) 
| bin _time span=1m
| stats Median(fill_perc) AS "fill_percentage" perc90(fill_perc) AS "90_perc" max(max) AS max max(curr) AS curr by host, _time, name 
| where (fill_percentage>70 AND name!="4 - Indexing Queue") OR (fill_percentage>70 AND name="4 - Indexing Queue")
| sort -_time

If you have 100% full queues, you have to check the network bandwidth occupation and eventually modify some configuration: e.g. In limits.conf of Universal  Forwarers

[thruput]
maxKBps = 0

Ciao.

Giuseppe