delayed logs

[splunkcol]

I have a problem with the logs, they are arriving with a delay of 12 hours or more

The information first reaches a syslog server and is forwarded to the indexers

When reviewing the logs in the syslog servers I find that they arrive without problem and with the correct date and time

when I go to the indexers or search heads to look at the logs I see that they have a delay of 12 hours or more

With this document I have tried to diagnose the problem but I cannot find the same panels that ask to review the document

in the part where it is suggested to check with the command iostat -zx 1 one of the parameters are in the values ​​cataloged as bad

https://www.splunk.com/pdfs/technical-briefs/disk-diagnosis-digging-deep-with-monitoring-console-and...

splunkcol_1-1601562510064.png

splunkcol_0-1601561959333.pngWhat else should I check?

splunkcol_2-1601562619818.png

[isoutamo]

How you are reading and forwarding those logs from syslog server? One issue could be that if/when you are using UF, you are hitting is't max default capacity?
This is good starting point for looking this issue: https://conf.splunk.com/files/2019/slides/FN1570.pdf
r. Ismo

[splunkcol]

yes, the syslog server receives the logs and forwards them to the indexers using UF

I understand that the cause of the queuing is typingqueue?

splunkcol_0-1601577492282.png

[isoutamo]

In your first message it shows that your disk io utilization is 100%. This means that it cannot handle more traffic without adding more disk to get more performance.
What kind of disk you have and what is amount of your daily/peak indexing volume?

[splunkcol]

In a previous post you suggested that I check that it will have a minimum IOPS, after checking, the disk has more than 800, it even has double. 

[isoutamo]

Splunk’s requirements is minimum 800 IOPS per disk to working. But it’s just minimum. Reality is totally dependent how much you are ingesting and what kind of query load you have. Here is link to reference hardware https://docs.splunk.com/Documentation/Splunk/8.0.6/Capacity/Referencehardware