All Apps and Add-ons

What are different types of alerts for detecting issues with Data Onboarding?

nm1984splunk
New Member

i am trying to figure out how we can detect issues with Data Onboarding. Is there any Splunk App available to identify issues with data onboarding ? what different alerts we can create in Splunk Deployment Server ?

Thanks.

0 Karma
1 Solution

gjanders
SplunkTrust
SplunkTrust

The list could go on forever depending on what problem your trying to resolve!

The main ones I have setup alerts for are:

  • Log stopped forwarding (Meta Woot!) helps here
  • Failure to parse timestamp correctly (when the event has not been broken part way through)
  • Future dated events
  • Insufficient permissions to read files
  • Universal forwarders exceeding the file descriptor cache
  • Universal forwarders having issues sending data to indexers
  • Universal forwarders that are time shifting
  • Universal forwarders failing due to lack of disk space
  • Universal forwarders with low ulimits
  • Time format changed multiple times in one sourcetype
  • Valid Timestamp Invalid Parsed Time
  • Weekly Broken Events Report
  • Weekly Truncated Logs Report

Each one of thse is an alert of some kind reporting on issues that I've found, if the post is upvoted I'm happy to share the ones you are interested in, some of them might help...

View solution in original post

gjanders
SplunkTrust
SplunkTrust

The list could go on forever depending on what problem your trying to resolve!

The main ones I have setup alerts for are:

  • Log stopped forwarding (Meta Woot!) helps here
  • Failure to parse timestamp correctly (when the event has not been broken part way through)
  • Future dated events
  • Insufficient permissions to read files
  • Universal forwarders exceeding the file descriptor cache
  • Universal forwarders having issues sending data to indexers
  • Universal forwarders that are time shifting
  • Universal forwarders failing due to lack of disk space
  • Universal forwarders with low ulimits
  • Time format changed multiple times in one sourcetype
  • Valid Timestamp Invalid Parsed Time
  • Weekly Broken Events Report
  • Weekly Truncated Logs Report

Each one of thse is an alert of some kind reporting on issues that I've found, if the post is upvoted I'm happy to share the ones you are interested in, some of them might help...

DalJeanis
SplunkTrust
SplunkTrust

That's really a broad question. There are answers all over this site about how to identify issues with servers not reporting, bad formatting, and a host of other issues.

In the big picture, splunk cannot know more about your data than you do, so your best practices are going to be to set up a load for one type of file at a time, set up the configuration in your sandbox server, test to make sure the data is being correctly interpreted, check to make sure that the volume of data is expected, then implement and verify all of the above on the production box. Then proceed to the next kind of data.

After that, any time that you perceive an issue with any particular kind of file, investigate, identify the issue, and set up alerts as needed to tell you when the problem re-occurs, for that kind of file and for any related type of file that is susceptible to the same issue.

It's a craft more than a science, and always remember the maxim - "Done is better than perfect."

adonio
Ultra Champion

can you kindly elaborate? you have pre-configured alerts on missing forwarders on DMC (MC).
What would you like to be alerted on?

0 Karma

nm1984splunk
New Member

Thanks adonio. I was checking about sending pre-configured alerts to ServiceNow for auto-ticketing.

Regards,
Nishant

0 Karma

adonio
Ultra Champion

in that case, I will recommend to use the add-on for service now and read the docs about how to integrate Splunk and SNOW here: https://docs.splunk.com/Documentation/AddOns/released/ServiceNow/Usescriptedalerts
and here:
https://docs.splunk.com/Documentation/AddOns/released/ServiceNow/Usecustomalertactions
there is plenty more.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...