Monitoring Splunk

How to troubleshoot issue with dmc_forwarder_assets.csv?

xouu
Explorer

Hi,
I receive alerts about : DMC Alert - Missing forwarders :

 

| inputlookup dmc_forwarder_assets
| search status="missing"
| rename hostname as Instance

 



it's telling me 3 forwarders are missing, old forwarders since the version is 8.2.1 and we are in 9.0.4.

 


If I search
| inputlookup dmc_forwarder_assets
then the result is good, and there are no missing instances anymore ! Where are the 3 missings forwarders actually coming from ?

Regarding the Rebuild forwarder assets in the MC does not change anything since the .csv
/opt/splunk/etc/apps/splunk_monitoring_console/lookups/dmc_forwarder_assets.csv is showing the right result.

My issue is the alerts reporting wrong informations.

Any ideas ? Ofc I can provide further informations.

Thanks


Labels (2)
0 Karma

xouu
Explorer

Hi @yeahnah Thanks for your reply

I did a search : 

(fwdType=* group=tcpin_connections guid=* index=_internal sourcetype=splunkd (connectionType=cooked OR connectionType=cookedSSL)) 

 

I got an error message : 
[splunk-ds, splunk-idx1, splunk-idx2] Could not load
lookup=LOOKUP-dmc_add_instance_info

0 Karma

yeahnah
Motivator

Hi @xouu 

That doesn't make sense as the base search does not have a lookup in it.   That error also indicates that the remote search peers do not have the look up definition but that should not prevent the search from completing.

I assume you are have the Splunk admin role and have access to the _internal indexes?

Are the servers in your Splunk Enterprise environment all at the version? 

0 Karma

xouu
Explorer

You're right, it's not related.

Yes, I am admin and all the splunk instances are in 9.0.4


0 Karma

yeahnah
Motivator

Hi @xouu 

The asset list output file is created from a schedeled report that runs every 30 mins, looking back 15min at _internal logs for UF messages.   You can review the DMC report here. 

https://<your_splunk_dmc_server>/en-GB/app/splunk_monitoring_console/report?s=%2FservicesNS%2Fnobody...

This is the reports's base search (v8.2.7) 

(fwdType=* group=tcpin_connections guid=* index=_internal sourcetype=splunkd (connectionType=cooked OR connectionType=cookedSSL)) 

The status="missing" logic is pretty simple.

| eval status=if(((isnull(sum_kb) OR (sum_kb <= 0)) OR (last_connected < (info_max_time - 900))),"missing","active")

If a forwarder log message matches any of the above expressions, it is marked as missing.

So, this is where you need to focus your investigation.  Why do these three forwarders fail the tests above. 

Hope that helps 

Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...