The universal forwarder (UF) seems to read the following files, but the files were not sent to the heavy forwarder (HF) around 11-26-2018 16:16. The following messages appeared in UF's splunkd.log around that time. It seems that Splunk read the files.
11-26-2018 16:16:16.109 +0900 INFO TailReader - Batch input finished reading file='/fxxxx/splunk/MMM2018092615.txt
11-26-2018 16:16:17.882 +0900 INFO TailReader - Batch input finished reading file='/fxxxx/splunk/OOO2018092615.txt
Following messages appear in UF's splunkd.log around the same time.
11-26-2018 16:16:05.005 +0900 INFO TcpOutputProc - Queue for group xxxxx_fwd_intermediate has stopped dropping events 11-26-2018 16:16:10.004 +0900 INFO TailReader - Could not send data to output queue (parsingQueue), retrying...
11-26-2018 16:16:10.005 +0900 WARN TcpOutputProc - Queue for group xxxxx_fwd_intermediate has begun dropping events
Could you tell me about solution?
Please check if the file size is large. For example, the size is 20-180MB, it seems that the default queue size of 500KB is really low for them. So, the queue got full is an expected behavior and increasing the queue size should be a solution for that.
UF's outputs.conf
[tcpout:xxxxx_fwd_intermediate]
maxQueueSize = 128MB
HF's inputs.conf
[splunktcp://9997]
disabled = 0
queueSize = 128MB