Getting Data In

Why is the substr function not working for JSON logs in Splunk 6.5.2?

New Member

The substr function is not working for json logs for us in 6.5.2 for Dev version. Whereas the Prod version of the Splunk seems to be supporting the same.

I checked the builds as well, both have the same. Log format is consistent across the two environments as well. Any clue as to what the case be here ?

0 Karma
1 Solution

SplunkTrust
SplunkTrust

Check if the msg field is a multivalued field as substr will not work on multivalued field.

If it is, something like this can be done,

index="web_rev3" _index_earliest=-15m@m _index_latest=now (level=ERROR OR level=FATAL) | eval Service=substr(index, 5)  | eval Time=strftime(_time, "%m/%d %H:%M:%S") | eval msg=if(mvcount(msg)>1, substr(mvindex(msg,0),1, 300), substr(msg,1,300)) | eval msg= msg . "..."  | table Time, Service,  msg | head 20

View solution in original post

SplunkTrust
SplunkTrust

Check if the msg field is a multivalued field as substr will not work on multivalued field.

If it is, something like this can be done,

index="web_rev3" _index_earliest=-15m@m _index_latest=now (level=ERROR OR level=FATAL) | eval Service=substr(index, 5)  | eval Time=strftime(_time, "%m/%d %H:%M:%S") | eval msg=if(mvcount(msg)>1, substr(mvindex(msg,0),1, 300), substr(msg,1,300)) | eval msg= msg . "..."  | table Time, Service,  msg | head 20

View solution in original post

New Member

Awesome ! Looks like this to be the issue. I will dig more into why we have multivalue fields as part of our logs.

0 Karma

SplunkTrust
SplunkTrust

Cool... you may want to look at this post as well to ensure this is not the case with yours

https://answers.splunk.com/answers/174939/why-are-my-json-fields-extracted-twice.html

0 Karma

SplunkTrust
SplunkTrust

"not working" covers a lot of ground. Please describe the symptoms as closely as possible, so we can help.

0 Karma

New Member

Yes, message field is very much there and it has valid content as well. Also field name casing is the same that I used as part of my query.

0 Karma

Contributor

OK so the msg field is present is it the substr that's failing or is it the concatenating of the '...' that's failing? What does the following produce?

 index="web_rev3" _index_earliest=-15m@m _index_latest=now (level=ERROR OR level=FATAL) 
 | head 1 | eval msg=substr(msg,1, 300) | table _time index msg
0 Karma

Esteemed Legend

I agree; strip down your search to the bare minimum failure stuff and get rid of all the cruft that is noise for us. Also, as a long shot, try running in verbose mode.

0 Karma

New Member

I have a field under the name of message that contains the content describing each of the log level. For the alert I have used the substr function to extract the first 300 characters of the message to avoid making email content heavy. However using the same just gives me a blank field. Below is the query to give you more idea.

index="web_rev3" _index_earliest=-15m@m _index_latest=now (level=ERROR OR level=FATAL) | eval Service=substr(index, 5)  | eval Time=strftime(_time, "%m/%d %H:%M:%S") | eval msg=substr(msg,1, 300) | eval msg= msg . "..."  | table Time, Service,  msg | head 20
0 Karma

Esteemed Legend

Have you checked that msg exists in both cases?

0 Karma

SplunkTrust
SplunkTrust

Okay, so first I'd answer woodcock's question - get rid of all the evals and see if msg exists

index="web_rev3" _index_earliest=-15m@m _index_latest=now (level=ERROR OR level=FATAL) 
| head 1 | table _time index msg

... if it doesn't exist, then try this and see if it's spelled some other way...

| head 1 | table _time index m*s*g* M*s*g*
0 Karma