forwarder troubleshoot dashboard

[Naa_Win]

Hello,

I’m working on creating a Splunk troubleshooting Dashboard for our internal team, who we are new to Splunk, to troubleshoot forwarder issues—specifically cases where no data is being received. I’d like to know the possible ways to troubleshoot forwarders when data is missing or for other related issues. Are there any existing dashboards I could use as a reference? also, what are the key metrics and internal index REST calls that I should focus on to cover all aspects of forwarder troubleshooting? 

#forwarder #troubleshoot #dashboard

[isoutamo]

Hi

here is one conf talk, How to find ingesting issues https://conf.splunk.com/files/2019/slides/FN1570.pdf.

There are many apps in splunkbase which helps you to find that kind of issues.

Also there are some conf presentations about this, but I cannot found those now 😞

r. Ismo

[isoutamo]

Here is the other one https://conf.splunk.com/files/2021/slides/PLA1410C.pdf

[gcusello]

Hi @Naa_Win ,

in all my projects I create a custom app containing dashboards to monitor infrastrcuture, with special attention to:

• fissing data sources,
• missing hosts,
• queues issues.

Ciao.

Giuseppe

[Naa_Win]

Hello @gcusello 

Thanks for the reply, is that possible to share the app info or share the source code of the dashboards ?

[gcusello]

Hi @Naa_Win ,

the dashboards depend on what you need:

if you need to see the hosts that sent logs in the last 30 days but not in the last hour, you can run:

| tstats count WHERE index=_internal earliest=-30d latest=now BY _time host
| where _time<now()-3600
| stats latest(_time) AS _time BY host

Then you can display the blocked queues and the status of queues using the searches that I shared at https://community.splunk.com/t5/Getting-Data-In/How-do-we-know-whether-typing-queues-are-blocked-or-...

and so on.

As I said they depend on what you need to display.

Ciao.

Giuseppe

[VatsalJagani]

There are few stuff that will be useful:

• You can use Monitoring Console's alert and dashboard
  • Dashboard -> Splunk Settings > Monitoring Console > Forwarders: Deployment
    • If setup has not done, then do the setup first (it will give you link to setup)
    • Alert -> Splunk Settings > Searches Reports & Alerts
      • Select App as Monitoring Console
      • Select Owner as All
      • And search for Missing Forwarder
      • Enable the alert -> "DMC Alert - Missing forwarders" and add your email to receive alerts on the email

There is one more search you can run to see what data forwarder is sending:

| tstats count where index=* host="<forwarder-host-name>" by index, sourcetype 

I hope this helps!!! Kindly upvote!!!

[Naa_Win]

Hello @VatsalJagani 

Thanks for the info, Yes we have those DMC enabled but the problem is as we are new to Splunk we had given only limited access for now to SH. So we wanted to create some dashboards to look with in the internal logs to detect the issues. I would like to start with the Universal Forwarder first.

[VatsalJagani]

That's why I suggested to look into DMC which has many searches. If you write those searches yourself it will take a lot of time. DMC will give those pre-built searches.

Now, if you don't have access to DMC in your environment, you can just install Splunk on your local laptop and use that to get searches.

To get the searches, you can open any panel in any panel, by clicking on the bottom-left "Open in search".

I hope this helps!!!