Alerting

Defining standards for using Splunk as an Error Handling tool

cwcoleman
Engager

I am working on a standards document for my team. It will define how best to use Splunk in regards to error handling.

I've started with the Splunk Logging Best Practices. http://dev.splunk.com/view/logging-best-practices/SP-CAAADP6

Is there any other documentation that anyone has created for this purpose? Mine will be directed towards our Java Development team. Although our Application Administrators (DEVOPS) will also be relying on it.

Notifications are a major topic I'm looking for advice on. Today we use some 'Alerts' that send email but they could be improved. Is there a best practice for creating logs that key alerts?

What about severity levels? Is this entirely application specific? Do you use a certain key=value that identifies the type of failure and write Alerts based on this data?

Thanks!

0 Karma
1 Solution

yannK
Splunk Employee
Splunk Employee

You want :

  1. a timestamp on the first line.
  2. key=values or key="value on a sentence" fields (easier to be automatically detected)
  3. fit in the defaults event parsing limits :
  • 10000 characters per line, and 255 lines per multiline events for the defaults
  • Otherwise, define a custom sourcetype with larger limits.

for severity, you can use the basic syslog DEBUG/INFO/WARN/ERROR/FATAL ...
or use an integers scale (that are easier to filter with a simple condition severity>2)

View solution in original post

yannK
Splunk Employee
Splunk Employee

You want :

  1. a timestamp on the first line.
  2. key=values or key="value on a sentence" fields (easier to be automatically detected)
  3. fit in the defaults event parsing limits :
  • 10000 characters per line, and 255 lines per multiline events for the defaults
  • Otherwise, define a custom sourcetype with larger limits.

for severity, you can use the basic syslog DEBUG/INFO/WARN/ERROR/FATAL ...
or use an integers scale (that are easier to filter with a simple condition severity>2)

cwcoleman
Engager

Great. These are all helpful items. I'll be adding each to my specification.

0 Karma
Get Updates on the Splunk Community!

Using Machine Learning for Hunting Security Threats

REGISTER NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more ...

Security Highlights | November 2022 Newsletter

 November 2022 2022 Gartner Magic Quadrant for SIEM: Splunk Named a Leader for the 9th Year in a RowSplunk is ...

Platform Highlights | November 2022 Newsletter

 November 2022 Skill Up on Splunk with our New Builder Tech Talk SeriesCan you build it? Yes you can! *play ...