Getting Data In

Why is the substr function not working for JSON logs in Splunk 6.5.2?

pimco_rgoyal
Observer

The substr function is not working for json logs for us in 6.5.2 for Dev version. Whereas the Prod version of the Splunk seems to be supporting the same.

I checked the builds as well, both have the same. Log format is consistent across the two environments as well. Any clue as to what the case be here ?

0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

Check if the msg field is a multivalued field as substr will not work on multivalued field.

If it is, something like this can be done,

index="web_rev3" _index_earliest=-15m@m _index_latest=now (level=ERROR OR level=FATAL) | eval Service=substr(index, 5)  | eval Time=strftime(_time, "%m/%d %H:%M:%S") | eval msg=if(mvcount(msg)>1, substr(mvindex(msg,0),1, 300), substr(msg,1,300)) | eval msg= msg . "..."  | table Time, Service,  msg | head 20

View solution in original post

somesoni2
SplunkTrust
SplunkTrust

Check if the msg field is a multivalued field as substr will not work on multivalued field.

If it is, something like this can be done,

index="web_rev3" _index_earliest=-15m@m _index_latest=now (level=ERROR OR level=FATAL) | eval Service=substr(index, 5)  | eval Time=strftime(_time, "%m/%d %H:%M:%S") | eval msg=if(mvcount(msg)>1, substr(mvindex(msg,0),1, 300), substr(msg,1,300)) | eval msg= msg . "..."  | table Time, Service,  msg | head 20

pimco_rgoyal
Observer

Awesome ! Looks like this to be the issue. I will dig more into why we have multivalue fields as part of our logs.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Cool... you may want to look at this post as well to ensure this is not the case with yours

https://answers.splunk.com/answers/174939/why-are-my-json-fields-extracted-twice.html

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

"not working" covers a lot of ground. Please describe the symptoms as closely as possible, so we can help.

0 Karma

pimco_rgoyal
Observer

Yes, message field is very much there and it has valid content as well. Also field name casing is the same that I used as part of my query.

0 Karma

davebrooking
Contributor

OK so the msg field is present is it the substr that's failing or is it the concatenating of the '...' that's failing? What does the following produce?

 index="web_rev3" _index_earliest=-15m@m _index_latest=now (level=ERROR OR level=FATAL) 
 | head 1 | eval msg=substr(msg,1, 300) | table _time index msg
0 Karma

woodcock
Esteemed Legend

I agree; strip down your search to the bare minimum failure stuff and get rid of all the cruft that is noise for us. Also, as a long shot, try running in verbose mode.

0 Karma

pimco_rgoyal
Observer

I have a field under the name of message that contains the content describing each of the log level. For the alert I have used the substr function to extract the first 300 characters of the message to avoid making email content heavy. However using the same just gives me a blank field. Below is the query to give you more idea.

index="web_rev3" _index_earliest=-15m@m _index_latest=now (level=ERROR OR level=FATAL) | eval Service=substr(index, 5)  | eval Time=strftime(_time, "%m/%d %H:%M:%S") | eval msg=substr(msg,1, 300) | eval msg= msg . "..."  | table Time, Service,  msg | head 20
0 Karma

woodcock
Esteemed Legend

Have you checked that msg exists in both cases?

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Okay, so first I'd answer woodcock's question - get rid of all the evals and see if msg exists

index="web_rev3" _index_earliest=-15m@m _index_latest=now (level=ERROR OR level=FATAL) 
| head 1 | table _time index msg

... if it doesn't exist, then try this and see if it's spelled some other way...

| head 1 | table _time index m*s*g* M*s*g*
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...