topic Re: splunk in All Apps and Add-ons

splunk

shakti — Mon, 06 May 2024 06:55:15 GMT

Hello ,

The Forwarder ingestion latency is showing red on my search head....

Root Cause(s):
- Indicator 'ingestion_latency_gap_multiplier' exceeded configured value. The observed value is 5474815. Message from 452CE67F-3C57-403C-B7B1-E34754172C83:10.250.2.7:3535

Can anyone please provide any suggestions?

Re: splunk

gcusello — Mon, 06 May 2024 09:02:53 GMT

Hi @shakti ,

there's a delay between the event timestamp and the indexing timestamp probably caused by the too high data volume.

This could be caused by a queue issue on the Forwarder, by a network latency or by a resource provlem (usually storage performance) on your Indexers.

You can check queues using a search like the following

index=_internal source=*metrics.log sourcetype=splunkd group=queue | eval name=case(name=="aggqueue","2 - Aggregation Queue", name=="indexqueue", "4 - Indexing Queue", name=="parsingqueue", "1 - Parsing Queue", name=="typingqueue", "3 - Typing Queue", name=="splunktcpin", "0 - TCP In Queue", name=="tcpin_cooked_pqueue", "0 - TCP In Queue") | eval max=if(isnotnull(max_size_kb),max_size_kb,max_size) | eval curr=if(isnotnull(current_size_kb),current_size_kb,current_size) | eval fill_perc=round((curr/max)*100,2) | bin _time span=1m | stats Median(fill_perc) AS "fill_percentage" perc90(fill_perc) AS "90_perc" max(max) AS max max(curr) AS curr by host, _time, name | where (fill_percentage>70 AND name!="4 - Indexing Queue") OR (fill_percentage>70 AND name="4 - Indexing Queue") | sort -_time

About resources, did you checked the IOPS of your storage?

have the correct number of CPUs?

at least, does your network have sufficient bandwidth to support your data volume?

Ciao.

Giuseppe

Re: splunk

shakti — Mon, 06 May 2024 09:07:13 GMT

@gcusello Appreciate your reply....

we have indexer clustering environment . However for both indexers and search head we are using only 4 CPU physical cores ..Do you think that can cause this problem?

Re: splunk

shakti — Mon, 06 May 2024 09:55:33 GMT

Also , if i may know what should be the good I/O operations for splunk?

Re: splunk

gcusello — Mon, 06 May 2024 10:34:03 GMT

Hi @shakti,

surely the number of CPUs in one of the root causes of your issue, because Splunk requires at least 12 CPUs for Indexers and 16 if you also have ES.

Anyway, check the IOPS (using a tool e.g. like Bonnie++ or FIO), because this is the usual major issue in queue problems.

Ciao.

Giuseppe

Re: splunk

shakti — Tue, 07 May 2024 07:02:34 GMT

@gcusello Thank you for your reply

The IOPS of indexers and search heads is between 50 -300 ...I guess its pretty less ...May I know do you have any suggestions how to improve on it?

Re: splunk

gcusello — Tue, 07 May 2024 07:07:44 GMT

Hi @shakti,

use faster disks if you're using a physical server or dedicated resources if you're using a virtual server and possible SSD disks.

Ciao.

Giuseppe