Getting Data In

Logging for Splunk - Best Practices or Tips?

shaneharter
New Member

Just getting started with Splunk. I'm looking to get better instrumentation and visibility into our systems. In some cases this could be for debugging, in other cases for audit-trail type uses.

Since I'm going to (re)write all of our logging code in these systems, does anybody have any suggestions?

Certainly one thing I'm looking to accomplish is "joining" events across log files as they flow through our system. Aside from that, my use cases seem pretty basic. But since i'm in this position where I can design my logs just for splunk I thought people who've been doing this for a while could have some tips that might make my life easier later on.

Thanks!
Shane

0 Karma
1 Solution

ziegfried
Influencer

This is a pretty good starting point: http://dev.splunk.com/view/logging-with-splunk/SP-CAAADP5

What I can add from my experience is that printing the following things in every event is really valuable:

  • The severity of the event (ie. DEBUG/INFO/WARN/ERROR)
  • A thread identifier (Some sequential or random string that identifies the current thread)
  • The source of the log event (ie. class, function or filename)

View solution in original post

ziegfried
Influencer

This is a pretty good starting point: http://dev.splunk.com/view/logging-with-splunk/SP-CAAADP5

What I can add from my experience is that printing the following things in every event is really valuable:

  • The severity of the event (ie. DEBUG/INFO/WARN/ERROR)
  • A thread identifier (Some sequential or random string that identifies the current thread)
  • The source of the log event (ie. class, function or filename)

Drainy
Champion

The identifier is a great point but missed by some and when combined with transaction you have a great tool for troubleshooting and debugging process/thread/event specific execution

0 Karma

mtevenan_splunk
Splunk Employee
Splunk Employee

The logging content referred to above is now here: http://dev.splunk.com/view/logging/SP-CAAAFCK

0 Karma
Get Updates on the Splunk Community!

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

See your relevant APM services, dashboards, and alerts in one place with the updated ...

As a Splunk Observability user, you have a lot of data you have to manage, prioritize, and troubleshoot on a ...

Index This | What goes away as soon as you talk about it?

May 2025 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this month’s ...