Why is heavy forwarder not sending _audit and _internal logs to indexer?

[neeravmathur]

Hi All,

We noticed that one of our Heavy Forwarder has not been sending _audit and _internal logs to our indexer. It is only sending _introspection and _telemetry logs to our Indexers. We need to have_audit/_internal logs from this server as well. Can anybody help?

 Telnet to the indexer works. Any help would be appreciated.

Running | tstats count where host=MYHF by index,_time only shows _introspection and _telemetry index

Thanks,

Neerav

[gcusello]

Hi @neeravmathur ,

check if you have logs using 

index=_internal (sourcetype=_audit OR sourcetype=_internal)

if yes, it's a visdualization problem, if not, you have some problem on the system, because Splunk doesn't send these logs when there's aproblem.

Ciao.

Giuseppe

[neeravmathur]

Hi gcusello,

No results for this search. Seems like a system issue then. Ideas for troubleshooting?? 

Thanks,

Neerav 

[gcusello]

Hi @neeravmathur ,

check if there's some queue on that HF, because _internal and _audit have a minor precedence than other logs.

You can find in Community many searches for queues, also from me.

If you don't find, I'll send it.

Ciao.

Giuseppe

[neeravmathur]

Hi, 

Is it something like the below search?

index=_internal group=queue | eval percfull=((current_size_kb/max_size_kb)*100) | search percfull>80 | dedup host, name | table _time host name current_size_kb max_size_kb

When running this on my search head I see my other heavy forwarders but the concerned HF is not returned in the results.

Also, there is a search which is running on all our heavy forwarders which writes data in summary index on our indexers. This HF is running the search and data is displayed but not ingested. 

Thanks,

Neerav

[gcusello · Accepted solution]

Hi @neeravmathur ,

by definition, you cannot run a search on an HF, only on Search Head.

On your SH, you have to run this search:

index=_internal  source=*metrics.log sourcetype=splunkd group=queue host=<your_HF_hostname> 
| eval name=case(name=="aggqueue","2 - Aggregation Queue",
 name=="indexqueue", "4 - Indexing Queue",
 name=="parsingqueue", "1 - Parsing Queue",
 name=="typingqueue", "3 - Typing Queue",
 name=="splunktcpin", "0 - TCP In Queue",
 name=="tcpin_cooked_pqueue", "0 - TCP In Queue") 
| eval max=if(isnotnull(max_size_kb),max_size_kb,max_size) 
| eval curr=if(isnotnull(current_size_kb),current_size_kb,current_size) 
| eval fill_perc=round((curr/max)*100,2) 
| bin _time span=1m
| stats Median(fill_perc) AS "fill_percentage" max(max) AS max max(curr) AS curr by host, _time, name 
| where (fill_percentage>70 AND name!="4 - Indexing Queue") OR (fill_percentage>70 AND name="4 - Indexing Queue")
| sort -_time

in this way you list the queues on your HF.

If you don't see it, check if it's correctly configured to send logs to Indexers.

Ciao.

Giuseppe

[neeravmathur]

Hi,

I am stupid. One of the guys had renamed the default host name for Index settings for this HF. So the logs were being forwarded under a different server name.

Should've been the first thing to check. Appreciate the help and thanks for your time !!!

Thanks,

Neerav

[gcusello]

Hi @neeravmathur ,

good for you.

if one answer solves your need, please accept one answer for the other people of Community or tell me how I can help you.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉