Key value pairs vs. JSON format and multiple key value pairs

[SramanJ]

Hello,

I am a new user to splunk and logging in general. So, appreciate your patience if my questions are fairly simple.

I am reviewing the Splunk best practices page and would like to get some opinions from expert Splunkers and community. We are building an application and we have some control over the format of the log events. We use Splunk (greater org) and I would like to make sure that the event format is best suited for our needs and also easier for Splunk to digest.

The best practices webpage (http://dev.splunk.com/view/logging-best-practices/SP-CAAADP6) lists use clear key value pairs and create events that are human readable. The best practice also suggests using developer friendly formats such as JSON and Paul’s blog explains how perl/shell scripts can be easily broken with key value pairs.

(1) What does Splunk work best with? In our case, We know that the number of key value pairs is not constant and will change across different kinds of events and same events. If number of key value pairs in an event is not constant, Does Splunk work better (** in terms of end user response time **) with JSON or Key value pairs or is it indifferent?

(2) Do users have to run the SPATH command to interpret events in JSON format? (or) Does Splunk does the interpretation (runs SPATH automatically when it sees JSON logs or customer configures SPATH once) automatically when it ingests logs in JSON format?

(3) Breaking up multi-value information: There is an example of multi-value information. We are going to have events in which multiple objects or applications will be involved. For example, A user can start multiple applications with one operation. Does the breaking up multi-value information best practice apply in this case? Which of the two approaches is the best?

Approach 1
Time=formatted time, Operation=start, app1=apache, app2= tomcat, eventid=xyzdfe324

(note that I have to use app1 and app2 to capture the fact that two distinct apps were started)

Approach 2
Time=formatted time, Operation=start, app=apache, eventid=xyzdfe324

Time=formatted time, Operation=start, app=apache, eventid=xyzdfe324

I will also be searching splunkbase, but I put together these questions so I am posting it anyway.

Thanks
SJ

[SramanJ]

Any response from experts/community?

Thanks