In a JSON Payload, parse out \n to newline

[skirven]

Hi! I'm trying to see if I can get a JSON Payload like this:

{"log":"2020-05-28 06:52:34,671 GMT TRACE [com.xxx.oss.core.servlets.TransactionFilter] (http-nio-8080-exec-7|R:lB6-JwrDGgR-ZvKy|ThreadId=55|ThreadPriority=5) Responding with outbound response: HTTP 200\\ncontent-length: 85\\ncontent-type: application/json\\n\\n{\"results\":true,\"internalTransactionId\":\"lB6-JwrDGgR-ZvKy\",\"executionTimeInMillis\":0}\n","container_name":"idm-geoservices","namespace_name":"dev","host":"server"}

To replace where I have a literal "\n", and present it as a New Line and also represent the JSON payload in the pretty JSON format, if possible.

I have tried something like:

| eval log=replace(log,"\\\\n","[\n]")

I can get the replacement to show when I do something like "TEST", but what I really want to do is have the "log" field present the \n in the message as a new line.

Longer term, we're going to implement Splunk Connect for Kubernetes, but we're trying to get our user taken care of with being able to parse out a multi-line JSON message from Kubernetes.

Thank you!
Stephen

[to4kawa]

| makeresults 
| eval _raw="{\"log\":\"2020-05-28 06:52:34,671 GMT TRACE [com.xxx.oss.core.servlets.TransactionFilter] (http-nio-8080-exec-7|R:lB6-JwrDGgR-ZvKy|ThreadId=55|ThreadPriority=5) Responding with outbound response: HTTP 200\\ncontent-length: 85\\ncontent-type: application/json\\n\\n{\"results\":true,\"internalTransactionId\":\"lB6-JwrDGgR-ZvKy\",\"executionTimeInMillis\":0}\n\",\"container_name\":\"idm-geoservices\",\"namespace_name\":\"dev\",\"host\":\"server\"}"
| rex "(?ms)log\":\"(?<log>.*?\})"
| eval log=split(replace(log,"(?ms)\\\n","#"),"#")
| rex mode=sed "s/{.*?}[^,]+,(.*)/{\1/"
| spath
| eval _raw=mvindex(log,4)
| spath

That's a lot of work.

[skirven]

Yes. That's a lot of work, and seems highly tailored to the one result. The problem I'm really just wanting to solve is to replace the "\n" with a normal regex \n and show the new line.

When the string was parsed with other data, it didn't represent the rest of the lines correctly. 😞
Thanks!
Stephen

[to4kawa]

| eval log=split(replace(log,"(?ms)\\\n","#"),"#")

If you want to solve the problem, see what the query is doing.
The respondent is not your subcontractor.

[Patrick_Peeters]

You can update the field with something like this:

<base search>| rex field=log mode=sed "s/\\\n/\n/g"

However that does not change the shown event, it only updates the field (visualise it with | table log for example).

To do it at index-time, make changes to props.conf like below:

[your_sourcetype]
SEDCMD-newline= s/\\\n/\n/g

[skirven]

@ppeeters_splunk - Hi! I actually have tried that too, and it didn't work.

I think this also mirrors the test case, and it's not parsing out what I was hoping. 😞

| makeresults | eval _raw="{\"log\":\"2020-05-28 06:52:34,671 GMT TRACE [com.xxx.oss.core.servlets.TransactionFilter] (http-nio-8080-exec-7|R:lB6-JwrDGgR-ZvKy|ThreadId=55|ThreadPriority=5) Responding with outbound response: HTTP 200\\ncontent-length: 85\\ncontent-type: application/json\\n\\n{\"results\":true,\"internalTransactionId\":\"lB6-JwrDGgR-ZvKy\",\"executionTimeInMillis\":0}\n\",\"container_name\":\"idm-geoservices\",\"namespace_name\":\"dev\",\"host\":\"server\"}" | rex field=log mode=sed "s/\\\n/\n/g"

[Patrick_Peeters]

Ah ok, in my case I had already used sourcetype=_json when indexing the data. In that case you can use | rex field=_raw mode=sed "s/\\\n/\n/g" as there is no log field.

It won't make it pretty JSON format though, just show the event slightly better. The issue you'll have is that the log field is not proper JSON in the first place, it's just a long string. You might get it to work with some regex trickery or in a complex way but you might be better off seeing if you can get the source data properly formatted first.

[javiergn · Accepted solution]

Hi @skirven,

If you just want to present that JSON in a nicer format then your only option is probably the makemv command:

| makemv delim="\\\\n" yourfield

As in the following screenshot. I know is not ideal, as you are creating a multivalue field, but it is presented in a much more readable way so hopefully it'll help.

Or you could also use spath to extract the fields and then apply makemv to log:

Let me know if that helps.

Thanks,
J

[skirven]

@javiergn - That actually didn't get anything parsed? 😞

[javiergn]

Sorry what didn't get anything parsed?
This?

| makemv delim="\\\\n" yourfield

What's your fieldname? Is it _raw? If so then use it like:

| makemv delim="\\\\n" _raw

You can also extract our JSON fields first with spath, as in the second screenshot, and then use makemv against the one containing the new lines (log in this case).

In any case, it always helps if you paste query here.

[skirven]

Looking at this solution closer, I actually proposed this solution to the user, and awaiting feedback. What I didn't think of was just dropping it into a table:

| makemv delim="\\n" log | table _time, log

[skirven]

This worked! Thanks!