topic Re: In a JSON Payload, parse out \n to newline in Splunk Search

In a JSON Payload, parse out \n to newline

skirven — Thu, 28 May 2020 07:32:42 GMT

Hi! I'm trying to see if I can get a JSON Payload like this:

{"log":"2020-05-28 06:52:34,671 GMT TRACE [com.xxx.oss.core.servlets.TransactionFilter] (http-nio-8080-exec-7|R:lB6-JwrDGgR-ZvKy|ThreadId=55|ThreadPriority=5) Responding with outbound response: HTTP 200\\ncontent-length: 85\\ncontent-type: application/json\\n\\n{\"results\":true,\"internalTransactionId\":\"lB6-JwrDGgR-ZvKy\",\"executionTimeInMillis\":0}\n","container_name":"idm-geoservices","namespace_name":"dev","host":"server"}

To replace where I have a literal "\n", and present it as a New Line and also represent the JSON payload in the pretty JSON format, if possible.

I have tried something like:

| eval log=replace(log,"\\\\n","[\n]")

I can get the replacement to show when I do something like "TEST", but what I really want to do is have the "log" field present the \n in the message as a new line.

Longer term, we're going to implement Splunk Connect for Kubernetes, but we're trying to get our user taken care of with being able to parse out a multi-line JSON message from Kubernetes.

Thank you!
Stephen

Re: In a JSON Payload, parse out \n to newline

javiergn — Thu, 28 May 2020 08:01:23 GMT

Hi @skirven,

If you just want to present that JSON in a nicer format then your only option is probably the makemv command:

| makemv delim="\\\\n" yourfield

As in the following screenshot. I know is not ideal, as you are creating a multivalue field, but it is presented in a much more readable way so hopefully it'll help.

Or you could also use spath to extract the fields and then apply makemv to log:

Let me know if that helps.

Thanks,
J

Re: In a JSON Payload, parse out \n to newline

Patrick_Peeters — Thu, 28 May 2020 08:01:39 GMT

You can update the field with something like this:

<base search>| rex field=log mode=sed "s/\\\n/\n/g"

However that does not change the shown event, it only updates the field (visualise it with | table log for example).

To do it at index-time, make changes to props.conf like below:

[your_sourcetype]
SEDCMD-newline= s/\\\n/\n/g

Re: In a JSON Payload, parse out \n to newline

skirven — Thu, 28 May 2020 08:19:10 GMT

@javiergn - That actually didn't get anything parsed? 😞

Re: In a JSON Payload, parse out \n to newline

skirven — Thu, 28 May 2020 08:20:36 GMT

@ppeeters_splunk - Hi! I actually have tried that too, and it didn't work.

I think this also mirrors the test case, and it's not parsing out what I was hoping. 😞

| makeresults | eval _raw="{\"log\":\"2020-05-28 06:52:34,671 GMT TRACE [com.xxx.oss.core.servlets.TransactionFilter] (http-nio-8080-exec-7|R:lB6-JwrDGgR-ZvKy|ThreadId=55|ThreadPriority=5) Responding with outbound response: HTTP 200\\ncontent-length: 85\\ncontent-type: application/json\\n\\n{\"results\":true,\"internalTransactionId\":\"lB6-JwrDGgR-ZvKy\",\"executionTimeInMillis\":0}\n\",\"container_name\":\"idm-geoservices\",\"namespace_name\":\"dev\",\"host\":\"server\"}" | rex field=log mode=sed "s/\\\n/\n/g"

Re: In a JSON Payload, parse out \n to newline

to4kawa — Thu, 28 May 2020 09:08:41 GMT

| makeresults 
| eval _raw="{\"log\":\"2020-05-28 06:52:34,671 GMT TRACE [com.xxx.oss.core.servlets.TransactionFilter] (http-nio-8080-exec-7|R:lB6-JwrDGgR-ZvKy|ThreadId=55|ThreadPriority=5) Responding with outbound response: HTTP 200\\ncontent-length: 85\\ncontent-type: application/json\\n\\n{\"results\":true,\"internalTransactionId\":\"lB6-JwrDGgR-ZvKy\",\"executionTimeInMillis\":0}\n\",\"container_name\":\"idm-geoservices\",\"namespace_name\":\"dev\",\"host\":\"server\"}"
| rex "(?ms)log\":\"(?<log>.*?\})"
| eval log=split(replace(log,"(?ms)\\\n","#"),"#")
| rex mode=sed "s/{.*?}[^,]+,(.*)/{\1/"
| spath
| eval _raw=mvindex(log,4)
| spath

That's a lot of work.

Re: In a JSON Payload, parse out \n to newline

javiergn — Thu, 28 May 2020 09:20:44 GMT

Sorry what didn't get anything parsed?
This?

| makemv delim="\\\\n" yourfield

What's your fieldname? Is it _raw? If so then use it like:

| makemv delim="\\\\n" _raw

You can also extract our JSON fields first with spath, as in the second screenshot, and then use makemv against the one containing the new lines (log in this case).

In any case, it always helps if you paste query here.

Re: In a JSON Payload, parse out \n to newline

skirven — Thu, 28 May 2020 10:06:39 GMT

Yes. That's a lot of work, and seems highly tailored to the one result. The problem I'm really just wanting to solve is to replace the "\n" with a normal regex \n and show the new line.

When the string was parsed with other data, it didn't represent the rest of the lines correctly. 😞
Thanks!
Stephen

Re: In a JSON Payload, parse out \n to newline

Patrick_Peeters — Thu, 28 May 2020 10:10:50 GMT

Ah ok, in my case I had already used sourcetype=_json when indexing the data. In that case you can use | rex field=_raw mode=sed "s/\\\n/\n/g" as there is no log field.

It won't make it pretty JSON format though, just show the event slightly better. The issue you'll have is that the log field is not proper JSON in the first place, it's just a long string. You might get it to work with some regex trickery or in a complex way but you might be better off seeing if you can get the source data properly formatted first.

Re: In a JSON Payload, parse out \n to newline

skirven — Thu, 28 May 2020 10:23:35 GMT

Looking at this solution closer, I actually proposed this solution to the user, and awaiting feedback. What I didn't think of was just dropping it into a table:

| makemv delim="\\n" log | table _time, log

Re: In a JSON Payload, parse out \n to newline

to4kawa — Thu, 28 May 2020 20:38:48 GMT

| eval log=split(replace(log,"(?ms)\\\n","#"),"#")

If you want to solve the problem, see what the query is doing.
The respondent is not your subcontractor.

Re: In a JSON Payload, parse out \n to newline

skirven — Thu, 28 May 2020 20:51:45 GMT

This worked! Thanks!