Why is the substr function not working for JSON logs in Splunk 6.5.2?

[pimco_rgoyal]

The substr function is not working for json logs for us in 6.5.2 for Dev version. Whereas the Prod version of the Splunk seems to be supporting the same.

I checked the builds as well, both have the same. Log format is consistent across the two environments as well. Any clue as to what the case be here ?

[somesoni2 · Accepted solution]

Check if the msg field is a multivalued field as substr will not work on multivalued field.

If it is, something like this can be done,

index="web_rev3" _index_earliest=-15m@m _index_latest=now (level=ERROR OR level=FATAL) | eval Service=substr(index, 5)  | eval Time=strftime(_time, "%m/%d %H:%M:%S") | eval msg=if(mvcount(msg)>1, substr(mvindex(msg,0),1, 300), substr(msg,1,300)) | eval msg= msg . "..."  | table Time, Service,  msg | head 20

[pimco_rgoyal]

Awesome ! Looks like this to be the issue. I will dig more into why we have multivalue fields as part of our logs.

[somesoni2]

Cool... you may want to look at this post as well to ensure this is not the case with yours

https://answers.splunk.com/answers/174939/why-are-my-json-fields-extracted-twice.html

[DalJeanis]

"not working" covers a lot of ground. Please describe the symptoms as closely as possible, so we can help.

[pimco_rgoyal]

Yes, message field is very much there and it has valid content as well. Also field name casing is the same that I used as part of my query.

[davebrooking]

OK so the msg field is present is it the substr that's failing or is it the concatenating of the '...' that's failing? What does the following produce?

 index="web_rev3" _index_earliest=-15m@m _index_latest=now (level=ERROR OR level=FATAL) 
 | head 1 | eval msg=substr(msg,1, 300) | table _time index msg

[woodcock]

I agree; strip down your search to the bare minimum failure stuff and get rid of all the cruft that is noise for us. Also, as a long shot, try running in verbose mode.

[pimco_rgoyal]

I have a field under the name of message that contains the content describing each of the log level. For the alert I have used the substr function to extract the first 300 characters of the message to avoid making email content heavy. However using the same just gives me a blank field. Below is the query to give you more idea.

index="web_rev3" _index_earliest=-15m@m _index_latest=now (level=ERROR OR level=FATAL) | eval Service=substr(index, 5)  | eval Time=strftime(_time, "%m/%d %H:%M:%S") | eval msg=substr(msg,1, 300) | eval msg= msg . "..."  | table Time, Service,  msg | head 20

[woodcock]

Have you checked that msg exists in both cases?

[DalJeanis]

Okay, so first I'd answer woodcock's question - get rid of all the evals and see if msg exists

index="web_rev3" _index_earliest=-15m@m _index_latest=now (level=ERROR OR level=FATAL) 
| head 1 | table _time index msg

... if it doesn't exist, then try this and see if it's spelled some other way...

| head 1 | table _time index m*s*g* M*s*g*