topic Re: Indexing Json objects to Splunk in Getting Data In

Indexing Json objects to Splunk

leustean — Tue, 19 Nov 2013 15:44:01 GMT

Hi all,

Until recently I used to print to standard output a single json object, effectively having it indexed into Splunk and it worked great for me. Each field in the Json object was correctly picked up by Splunk and the Json object was turned into an event.

My props.conf looks like :

[default]
KV_MODE = json
LINE_BREAKER = "(^){"
NO_BINARY_CHECK = 1
TRUNCATE = 0
SHOULD_LINEMERGE = false    

[my-source]
DATETIME_CONFIG = CURRENT

But since I needed to extend the functionality I began printing in a loop several json objects :

for st in stats:            
   # Index each json object to Splunk
    print (json.dumps(st))
    sys.stdout.flush()

The effect is that now all json objects are mashed up together in a single event no field is detected .

Could someone provide input on how to have every Json Object in a separately Event ?

Cheers

Re: Indexing Json objects to Splunk

lguinn2 — Mon, 28 Sep 2020 15:19:07 GMT

I would do it this way:

[my-source]
DATETIME_CONFIG = CURRENT
KV_MODE = json
NO_BINARY_CHECK = 1
TRUNCATE = 0
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = someregularexpression
MAX_EVENTS = 1000

First, I would not put anything in the default stanza unless you mean for it to apply to every input forever.

Second, the LINE_BREAKER is longer useful, and Splunk does need to line-merge events. So to define the split between events, you need to tell Splunk what to look for. I usually use BREAK_ONLY_BEFORE, which should be set to a regular expression that matches a string that will only (and always) appear on the first line of each JSON event. Note that regular expression can appear anywhere within the first line - it doesn't have to be at the beginning - and Splunk will still break at the beginning of the line. Finally, MAX_EVENTS is not really the maximum number of events - it is the maximum number of lines in an event. So I made it larger than the default of 256, but that might not be necessary for you.

In the manual, you might want to look at the event processing section.

Re: Indexing Json objects to Splunk

leustean — Wed, 20 Nov 2013 08:53:07 GMT

For me removing the LINE_BREAKER = "(^){" from the props.conf file did the trick