Getting Data In

delayed logs

splunkcol
Builder

I have a problem with the logs, they are arriving with a delay of 12 hours or more

The information first reaches a syslog server and is forwarded to the indexers

When reviewing the logs in the syslog servers I find that they arrive without problem and with the correct date and time

when I go to the indexers or search heads to look at the logs I see that they have a delay of 12 hours or more

 

With this document I have tried to diagnose the problem but I cannot find the same panels that ask to review the document

in the part where it is suggested to check with the command iostat -zx 1 one of the parameters are in the values ​​cataloged as bad

https://www.splunk.com/pdfs/technical-briefs/disk-diagnosis-digging-deep-with-monitoring-console-and...

splunkcol_1-1601562510064.png

splunkcol_0-1601561959333.pngWhat else should I check?

splunkcol_2-1601562619818.png

 

 

 

Labels (2)
0 Karma

isoutamo
SplunkTrust
SplunkTrust
How you are reading and forwarding those logs from syslog server? One issue could be that if/when you are using UF, you are hitting is't max default capacity?
This is good starting point for looking this issue: https://conf.splunk.com/files/2019/slides/FN1570.pdf
r. Ismo
0 Karma

splunkcol
Builder

yes, the syslog server receives the logs and forwards them to the indexers using UF

I understand that the cause of the queuing is typingqueue?

 

splunkcol_0-1601577492282.png

 

0 Karma

isoutamo
SplunkTrust
SplunkTrust
In your first message it shows that your disk io utilization is 100%. This means that it cannot handle more traffic without adding more disk to get more performance.
What kind of disk you have and what is amount of your daily/peak indexing volume?
0 Karma

splunkcol
Builder

In a previous post you suggested that I check that it will have a minimum IOPS, after checking, the disk has more than 800, it even has double. 

0 Karma

isoutamo
SplunkTrust
SplunkTrust
Splunk’s requirements is minimum 800 IOPS per disk to working. But it’s just minimum. Reality is totally dependent how much you are ingesting and what kind of query load you have. Here is link to reference hardware https://docs.splunk.com/Documentation/Splunk/8.0.6/Capacity/Referencehardware
0 Karma
Get Updates on the Splunk Community!

Fueling your curiosity with new Splunk ILT and eLearning courses

At Splunk Education, we’re driven by curiosity—both ours and yours! That’s why we’re committed to delivering ...

Splunk AI Assistant for SPL 1.1.0 | Now Personalized to Your Environment for Greater ...

Splunk AI Assistant for SPL has transformed how users interact with Splunk, making it easier than ever to ...

Unleash Unified Security and Observability with Splunk Cloud Platform

     Now Available on Microsoft AzureOn Demand Now Step boldly into the AI revolution with enhanced security ...