How to troubleshoot issue with dmc_forwarder_assets.csv?

[xouu]

Hi,
I receive alerts about : DMC Alert - Missing forwarders :

| inputlookup dmc_forwarder_assets
| search status="missing"
| rename hostname as Instance

it's telling me 3 forwarders are missing, old forwarders since the version is 8.2.1 and we are in 9.0.4.

If I search
| inputlookup dmc_forwarder_assets
then the result is good, and there are no missing instances anymore ! Where are the 3 missings forwarders actually coming from ?

Regarding the Rebuild forwarder assets in the MC does not change anything since the .csv
/opt/splunk/etc/apps/splunk_monitoring_console/lookups/dmc_forwarder_assets.csv is showing the right result.

My issue is the alerts reporting wrong informations.

Any ideas ? Ofc I can provide further informations.

Thanks

[xouu]

Hi @yeahnah Thanks for your reply

I did a search : 

(fwdType=* group=tcpin_connections guid=* index=_internal sourcetype=splunkd (connectionType=cooked OR connectionType=cookedSSL)) 

I got an error message : 
[splunk-ds, splunk-idx1, splunk-idx2] Could not load
lookup=LOOKUP-dmc_add_instance_info

[yeahnah]

Hi @xouu 

That doesn't make sense as the base search does not have a lookup in it.   That error also indicates that the remote search peers do not have the look up definition but that should not prevent the search from completing.

I assume you are have the Splunk admin role and have access to the _internal indexes?

Are the servers in your Splunk Enterprise environment all at the version? 

[xouu]

You're right, it's not related.

Yes, I am admin and all the splunk instances are in 9.0.4

[yeahnah]

Hi @xouu 

The asset list output file is created from a schedeled report that runs every 30 mins, looking back 15min at _internal logs for UF messages.   You can review the DMC report here. 

https://<your_splunk_dmc_server>/en-GB/app/splunk_monitoring_console/report?s=%2FservicesNS%2Fnobody...

This is the reports's base search (v8.2.7) 

(fwdType=* group=tcpin_connections guid=* index=_internal sourcetype=splunkd (connectionType=cooked OR connectionType=cookedSSL)) 

The status="missing" logic is pretty simple.

| eval status=if(((isnull(sum_kb) OR (sum_kb <= 0)) OR (last_connected < (info_max_time - 900))),"missing","active")

If a forwarder log message matches any of the above expressions, it is marked as missing.

So, this is where you need to focus your investigation.  Why do these three forwarders fail the tests above. 

Hope that helps