topic Re: forwarder troubleshoot dashboard in Dashboards & Visualizations

forwarder troubleshoot dashboard

Naa_Win — Tue, 31 Dec 2024 03:28:12 GMT

Hello,

I’m working on creating a Splunk troubleshooting Dashboard for our internal team, who we are new to Splunk, to troubleshoot forwarder issues—specifically cases where no data is being received. I’d like to know the possible ways to troubleshoot forwarders when data is missing or for other related issues. Are there any existing dashboards I could use as a reference? also, what are the key metrics and internal index REST calls that I should focus on to cover all aspects of forwarder troubleshooting?

#forwarder #troubleshoot #dashboard

Re: forwarder troubleshoot dashboard

VatsalJagani — Tue, 31 Dec 2024 15:06:58 GMT

There are few stuff that will be useful:

You can use Monitoring Console's alert and dashboard
- Dashboard -> Splunk Settings > Monitoring Console > Forwarders: Deployment
  - If setup has not done, then do the setup first (it will give you link to setup)
  - Alert -> Splunk Settings > Searches Reports & Alerts
    - Select App as Monitoring Console
    - Select Owner as All
    - And search for Missing Forwarder
    - Enable the alert -> "DMC Alert - Missing forwarders" and add your email to receive alerts on the email

There is one more search you can run to see what data forwarder is sending:

| tstats count where index=* host="<forwarder-host-name>" by index, sourcetype

I hope this helps!!! Kindly upvote!!!

Re: forwarder troubleshoot dashboard

gcusello — Tue, 31 Dec 2024 07:44:59 GMT

Hi @Naa_Win ,

in all my projects I create a custom app containing dashboards to monitor infrastrcuture, with special attention to:

fissing data sources,
missing hosts,
queues issues.

Ciao.

Giuseppe

Re: forwarder troubleshoot dashboard

isoutamo — Tue, 31 Dec 2024 09:02:33 GMT

here is one conf talk, How to find ingesting issues https://conf.splunk.com/files/2019/slides/FN1570.pdf.

There are many apps in splunkbase which helps you to find that kind of issues.

Also there are some conf presentations about this, but I cannot found those now 😞

r. Ismo

Re: forwarder troubleshoot dashboard

Naa_Win — Fri, 03 Jan 2025 14:35:12 GMT

Hello @gcusello

Thanks for the reply, is that possible to share the app info or share the source code of the dashboards ?

Re: forwarder troubleshoot dashboard

Naa_Win — Fri, 03 Jan 2025 14:38:04 GMT

Hello @VatsalJagani

Thanks for the info, Yes we have those DMC enabled but the problem is as we are new to Splunk we had given only limited access for now to SH. So we wanted to create some dashboards to look with in the internal logs to detect the issues. I would like to start with the Universal Forwarder first.

Re: forwarder troubleshoot dashboard

VatsalJagani — Fri, 03 Jan 2025 15:35:16 GMT

That's why I suggested to look into DMC which has many searches. If you write those searches yourself it will take a lot of time. DMC will give those pre-built searches.

Now, if you don't have access to DMC in your environment, you can just install Splunk on your local laptop and use that to get searches.

To get the searches, you can open any panel in any panel, by clicking on the bottom-left "Open in search".

I hope this helps!!!

Re: forwarder troubleshoot dashboard

gcusello — Sat, 04 Jan 2025 09:05:52 GMT

Hi @Naa_Win ,

the dashboards depend on what you need:

if you need to see the hosts that sent logs in the last 30 days but not in the last hour, you can run:

| tstats count WHERE index=_internal earliest=-30d latest=now BY _time host | where _time<now()-3600 | stats latest(_time) AS _time BY host

Then you can display the blocked queues and the status of queues using the searches that I shared at https://community.splunk.com/t5/Getting-Data-In/How-do-we-know-whether-typing-queues-are-blocked-or-not/m-p/586347

and so on.

As I said they depend on what you need to display.

Ciao.

Giuseppe

Re: forwarder troubleshoot dashboard

isoutamo — Wed, 08 Jan 2025 19:55:41 GMT

Here is the other one https://conf.splunk.com/files/2021/slides/PLA1410C.pdf