Getting Data In

delayed logs

splunkcol
Builder

I have a problem with the logs, they are arriving with a delay of 12 hours or more

The information first reaches a syslog server and is forwarded to the indexers

When reviewing the logs in the syslog servers I find that they arrive without problem and with the correct date and time

when I go to the indexers or search heads to look at the logs I see that they have a delay of 12 hours or more

 

With this document I have tried to diagnose the problem but I cannot find the same panels that ask to review the document

in the part where it is suggested to check with the command iostat -zx 1 one of the parameters are in the values ​​cataloged as bad

https://www.splunk.com/pdfs/technical-briefs/disk-diagnosis-digging-deep-with-monitoring-console-and...

splunkcol_1-1601562510064.png

splunkcol_0-1601561959333.pngWhat else should I check?

splunkcol_2-1601562619818.png

 

 

 

Labels (3)
0 Karma

isoutamo
SplunkTrust
SplunkTrust
How you are reading and forwarding those logs from syslog server? One issue could be that if/when you are using UF, you are hitting is't max default capacity?
This is good starting point for looking this issue: https://conf.splunk.com/files/2019/slides/FN1570.pdf
r. Ismo
0 Karma

splunkcol
Builder

yes, the syslog server receives the logs and forwards them to the indexers using UF

I understand that the cause of the queuing is typingqueue?

 

splunkcol_0-1601577492282.png

 

0 Karma

isoutamo
SplunkTrust
SplunkTrust
In your first message it shows that your disk io utilization is 100%. This means that it cannot handle more traffic without adding more disk to get more performance.
What kind of disk you have and what is amount of your daily/peak indexing volume?
0 Karma

splunkcol
Builder

In a previous post you suggested that I check that it will have a minimum IOPS, after checking, the disk has more than 800, it even has double. 

0 Karma

isoutamo
SplunkTrust
SplunkTrust
Splunk’s requirements is minimum 800 IOPS per disk to working. But it’s just minimum. Reality is totally dependent how much you are ingesting and what kind of query load you have. Here is link to reference hardware https://docs.splunk.com/Documentation/Splunk/8.0.6/Capacity/Referencehardware
0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...