Logging for Splunk - Best Practices or Tips?

[shaneharter]

Just getting started with Splunk. I'm looking to get better instrumentation and visibility into our systems. In some cases this could be for debugging, in other cases for audit-trail type uses.

Since I'm going to (re)write all of our logging code in these systems, does anybody have any suggestions?

Certainly one thing I'm looking to accomplish is "joining" events across log files as they flow through our system. Aside from that, my use cases seem pretty basic. But since i'm in this position where I can design my logs just for splunk I thought people who've been doing this for a while could have some tips that might make my life easier later on.

Thanks!
Shane

[ziegfried · Accepted solution]

This is a pretty good starting point: http://dev.splunk.com/view/logging-with-splunk/SP-CAAADP5

What I can add from my experience is that printing the following things in every event is really valuable:

• The severity of the event (ie. DEBUG/INFO/WARN/ERROR)
• A thread identifier (Some sequential or random string that identifies the current thread)
• The source of the log event (ie. class, function or filename)

[Drainy]

The identifier is a great point but missed by some and when combined with transaction you have a great tool for troubleshooting and debugging process/thread/event specific execution

[mtevenan_splunk]

The logging content referred to above is now here: http://dev.splunk.com/view/logging/SP-CAAAFCK