topic Heavy forwarder connections to Splunk indexers by checking internal logs for any type of issues. in Getting Data In

Heavy forwarder connections to Splunk indexers by checking internal logs for any type of issues.

phanikumarcs — Fri, 02 Aug 2024 07:48:08 GMT

i have to create an alert to monitor any issue happens for HF to Indexers, by checking internal logs. I am using this spl. Need suggestions or correct SPL.

index=_internal source=*metrics.log group=tcpin_connections hostname="*hf*"

Re: Heavy forwarder connections to Splunk indexers by checking internal logs for any type of issues.

gcusello — Fri, 02 Aug 2024 08:04:19 GMT

Hi @phanikumarcs ,

I'd simplify your search to search eventual erros:

index=_internal hostname="*hf*"

whot do you want to monitor?

Ciao.

Giuseppe

Re: Heavy forwarder connections to Splunk indexers by checking internal logs for any type of issues.

phanikumarcs — Fri, 02 Aug 2024 08:07:08 GMT

Yes, but we have to setup a alert for example if any issues will get based on warns or errors it will trigger

Re: Heavy forwarder connections to Splunk indexers by checking internal logs for any type of issues.

gcusello — Fri, 02 Aug 2024 08:16:18 GMT

Hi @phanikumarcs ,

as I said, define your Use Cases, then you could create your searches.

e.g. you could create an alert for the queues:

index=_internal source=*metrics.log sourcetype=splunkd group=queue | eval name=case(name=="aggqueue","2 - Aggregation Queue", name=="indexqueue", "4 - Indexing Queue", name=="parsingqueue", "1 - Parsing Queue", name=="typingqueue", "3 - Typing Queue", name=="splunktcpin", "0 - TCP In Queue", name=="tcpin_cooked_pqueue", "0 - TCP In Queue") | eval max=if(isnotnull(max_size_kb),max_size_kb,max_size) | eval curr=if(isnotnull(current_size_kb),current_size_kb,current_size) | eval fill_perc=round((curr/max)*100,2) | bin _time span=1m | stats Median(fill_perc) AS "fill_percentage" perc90(fill_perc) AS "90_perc" max(max) AS max max(curr) AS curr by host, _time, name | where (fill_percentage>70 AND name!="4 - Indexing Queue") OR (fill_percentage>70 AND name="4 - Indexing Queue") | sort -_time

then you could check the disk space,

or whtelse you like.

Anyway: define your Use Cases

Ciao.

Giuseppe