Archive
Highlighted

What are different types of alerts for detecting issues with Data Onboarding?

New Member

i am trying to figure out how we can detect issues with Data Onboarding. Is there any Splunk App available to identify issues with data onboarding ? what different alerts we can create in Splunk Deployment Server ?

Thanks.

0 Karma
Highlighted

Re: What are different types of alerts for detecting issues with Data Onboarding?

SplunkTrust
SplunkTrust

can you kindly elaborate? you have pre-configured alerts on missing forwarders on DMC (MC).
What would you like to be alerted on?

0 Karma
Highlighted

Re: What are different types of alerts for detecting issues with Data Onboarding?

New Member

Thanks adonio. I was checking about sending pre-configured alerts to ServiceNow for auto-ticketing.

Regards,
Nishant

0 Karma
Highlighted

Re: What are different types of alerts for detecting issues with Data Onboarding?

SplunkTrust
SplunkTrust

in that case, I will recommend to use the add-on for service now and read the docs about how to integrate Splunk and SNOW here: https://docs.splunk.com/Documentation/AddOns/released/ServiceNow/Usescriptedalerts
and here:
https://docs.splunk.com/Documentation/AddOns/released/ServiceNow/Usecustomalertactions
there is plenty more.

0 Karma
Highlighted

Re: What are different types of alerts for detecting issues with Data Onboarding?

SplunkTrust
SplunkTrust

That's really a broad question. There are answers all over this site about how to identify issues with servers not reporting, bad formatting, and a host of other issues.

In the big picture, splunk cannot know more about your data than you do, so your best practices are going to be to set up a load for one type of file at a time, set up the configuration in your sandbox server, test to make sure the data is being correctly interpreted, check to make sure that the volume of data is expected, then implement and verify all of the above on the production box. Then proceed to the next kind of data.

After that, any time that you perceive an issue with any particular kind of file, investigate, identify the issue, and set up alerts as needed to tell you when the problem re-occurs, for that kind of file and for any related type of file that is susceptible to the same issue.

It's a craft more than a science, and always remember the maxim - "Done is better than perfect."

Highlighted

Re: What are different types of alerts for detecting issues with Data Onboarding?

SplunkTrust
SplunkTrust

The list could go on forever depending on what problem your trying to resolve!

The main ones I have setup alerts for are:

  • Log stopped forwarding (Meta Woot!) helps here
  • Failure to parse timestamp correctly (when the event has not been broken part way through)
  • Future dated events
  • Insufficient permissions to read files
  • Universal forwarders exceeding the file descriptor cache
  • Universal forwarders having issues sending data to indexers
  • Universal forwarders that are time shifting
  • Universal forwarders failing due to lack of disk space
  • Universal forwarders with low ulimits
  • Time format changed multiple times in one sourcetype
  • Valid Timestamp Invalid Parsed Time
  • Weekly Broken Events Report
  • Weekly Truncated Logs Report

Each one of thse is an alert of some kind reporting on issues that I've found, if the post is upvoted I'm happy to share the ones you are interested in, some of them might help...

View solution in original post