topic Re: delayed logs in Getting Data In

delayed logs

splunkcol — Thu, 01 Oct 2020 14:34:44 GMT

I have a problem with the logs, they are arriving with a delay of 12 hours or more

The information first reaches a syslog server and is forwarded to the indexers

When reviewing the logs in the syslog servers I find that they arrive without problem and with the correct date and time

when I go to the indexers or search heads to look at the logs I see that they have a delay of 12 hours or more

With this document I have tried to diagnose the problem but I cannot find the same panels that ask to review the document

in the part where it is suggested to check with the command iostat -zx 1 one of the parameters are in the values cataloged as bad

https://www.splunk.com/pdfs/technical-briefs/disk-diagnosis-digging-deep-with-monitoring-console-and-more.pdf

What else should I check?

Re: delayed logs

isoutamo — Thu, 01 Oct 2020 14:39:26 GMT

How you are reading and forwarding those logs from syslog server? One issue could be that if/when you are using UF, you are hitting is't max default capacity?
This is good starting point for looking this issue: https://conf.splunk.com/files/2019/slides/FN1570.pdf
r. Ismo

Re: delayed logs

splunkcol — Thu, 01 Oct 2020 18:42:41 GMT

yes, the syslog server receives the logs and forwards them to the indexers using UF

I understand that the cause of the queuing is typingqueue?

Re: delayed logs

isoutamo — Thu, 01 Oct 2020 19:02:15 GMT

In your first message it shows that your disk io utilization is 100%. This means that it cannot handle more traffic without adding more disk to get more performance.
What kind of disk you have and what is amount of your daily/peak indexing volume?

Re: delayed logs

splunkcol — Thu, 01 Oct 2020 19:12:16 GMT

In a previous post you suggested that I check that it will have a minimum IOPS, after checking, the disk has more than 800, it even has double.

Re: delayed logs

isoutamo — Thu, 01 Oct 2020 19:21:19 GMT

Splunk’s requirements is minimum 800 IOPS per disk to working. But it’s just minimum. Reality is totally dependent how much you are ingesting and what kind of query load you have. Here is link to reference hardware https://docs.splunk.com/Documentation/Splunk/8.0.6/Capacity/Referencehardware