topic Why is the substr function not working for JSON logs in Splunk 6.5.2? in Getting Data In

Why is the substr function not working for JSON logs in Splunk 6.5.2?

pimco_rgoyal — Wed, 14 Jun 2017 20:49:25 GMT

The substr function is not working for json logs for us in 6.5.2 for Dev version. Whereas the Prod version of the Splunk seems to be supporting the same.

I checked the builds as well, both have the same. Log format is consistent across the two environments as well. Any clue as to what the case be here ?

Re: Why is the substr function not working for JSON logs in Splunk 6.5.2?

DalJeanis — Thu, 15 Jun 2017 01:41:14 GMT

"not working" covers a lot of ground. Please describe the symptoms as closely as possible, so we can help.

Re: Why is the substr function not working for JSON logs in Splunk 6.5.2?

pimco_rgoyal — Thu, 15 Jun 2017 05:50:05 GMT

I have a field under the name of message that contains the content describing each of the log level. For the alert I have used the substr function to extract the first 300 characters of the message to avoid making email content heavy. However using the same just gives me a blank field. Below is the query to give you more idea.

index="web_rev3" _index_earliest=-15m@m _index_latest=now (level=ERROR OR level=FATAL) | eval Service=substr(index, 5)  | eval Time=strftime(_time, "%m/%d %H:%M:%S") | eval msg=substr(msg,1, 300) | eval msg= msg . "..."  | table Time, Service,  msg | head 20

Re: Why is the substr function not working for JSON logs in Splunk 6.5.2?

woodcock — Fri, 16 Jun 2017 02:18:49 GMT

Have you checked that msg exists in both cases?

Re: Why is the substr function not working for JSON logs in Splunk 6.5.2?

DalJeanis — Fri, 16 Jun 2017 03:04:13 GMT

Okay, so first I'd answer woodcock's question - get rid of all the evals and see if msg exists

index="web_rev3" _index_earliest=-15m@m _index_latest=now (level=ERROR OR level=FATAL) 
| head 1 | table _time index msg

... if it doesn't exist, then try this and see if it's spelled some other way...

| head 1 | table _time index m*s*g* M*s*g*

Re: Why is the substr function not working for JSON logs in Splunk 6.5.2?

pimco_rgoyal — Fri, 16 Jun 2017 06:24:32 GMT

Yes, message field is very much there and it has valid content as well. Also field name casing is the same that I used as part of my query.

Re: Why is the substr function not working for JSON logs in Splunk 6.5.2?

davebrooking — Fri, 16 Jun 2017 09:43:30 GMT

OK so the msg field is present is it the substr that's failing or is it the concatenating of the '...' that's failing? What does the following produce?

 index="web_rev3" _index_earliest=-15m@m _index_latest=now (level=ERROR OR level=FATAL) 
 | head 1 | eval msg=substr(msg,1, 300) | table _time index msg

Re: Why is the substr function not working for JSON logs in Splunk 6.5.2?

woodcock — Fri, 16 Jun 2017 15:13:37 GMT

I agree; strip down your search to the bare minimum failure stuff and get rid of all the cruft that is noise for us. Also, as a long shot, try running in verbose mode.

Re: Why is the substr function not working for JSON logs in Splunk 6.5.2?

somesoni2 — Fri, 16 Jun 2017 17:34:36 GMT

Check if the msg field is a multivalued field as substr will not work on multivalued field.

If it is, something like this can be done,

index="web_rev3" _index_earliest=-15m@m _index_latest=now (level=ERROR OR level=FATAL) | eval Service=substr(index, 5)  | eval Time=strftime(_time, "%m/%d %H:%M:%S") | eval msg=if(mvcount(msg)>1, substr(mvindex(msg,0),1, 300), substr(msg,1,300)) | eval msg= msg . "..."  | table Time, Service,  msg | head 20

Re: Why is the substr function not working for JSON logs in Splunk 6.5.2?

pimco_rgoyal — Fri, 16 Jun 2017 17:40:54 GMT

Awesome ! Looks like this to be the issue. I will dig more into why we have multivalue fields as part of our logs.

Re: Why is the substr function not working for JSON logs in Splunk 6.5.2?

somesoni2 — Fri, 16 Jun 2017 17:44:09 GMT

Cool... you may want to look at this post as well to ensure this is not the case with yours

https://answers.splunk.com/answers/174939/why-are-my-json-fields-extracted-twice.html