topic Re: Assistance Needed for Identifying Logs for Connection Issues in Splunk in Getting Data In

Assistance Needed for Identifying Logs for Connection Issues in Splunk

Rahul_a — Wed, 12 Jun 2024 16:56:26 GMT

I am planning to build a Splunk dashboard for monitoring connection issues from various sources. Specifically, I need to identify when a connection fails or when an application stops sending data to Splunk and display these issues on the dashboard. The data sources include:

Application server universal forwarder to our Splunk heavy forwarder
HEC (HTTP Event Collector)
Various add-ons (e.g., Azure add-on, AWS add-on, DB Connect add-on)

I am aware that many logs can be found under index=_internal, but I need assistance in identifying the necessary logs that pertain to real-time errors or connection failures. Could you please help me with this?

Re: Assistance Needed for Identifying Logs for Connection Issues in Splunk

Rahul_a — Thu, 27 Jun 2024 22:21:12 GMT

Hello
I know its already 2 weeks but still waiting for answer can any one help me out

Re: Assistance Needed for Identifying Logs for Connection Issues in Splunk

PickleRick — Tue, 02 Jul 2024 07:46:09 GMT

There are at least two separate issues here.

One is monitoring for data that used to be ingested but is no more, regardless of the reason for it (maybe there is a configuration problem on the receving end, maybe the source simply stopped sending data, maybe something else). There are several apps for that on Splunkbase. For example TrackMe - https://splunkbase.splunk.com/app/4621

Another thing is finding errors coming from your inputs (expired certs, broken connections, non-responding API endpoints and so on). And this is something you'd normally look for in _internal index indeed add those you'll find primarily in splunkd.log but also specific add-ons can create their own log files. So it's a bit more complicated than just a single search to find everything that's wrong.