Splunk Search

In a JSON Payload, parse out \n to newline

skirven
Communicator

Hi! I'm trying to see if I can get a JSON Payload like this:

{"log":"2020-05-28 06:52:34,671 GMT TRACE [com.xxx.oss.core.servlets.TransactionFilter] (http-nio-8080-exec-7|R:lB6-JwrDGgR-ZvKy|ThreadId=55|ThreadPriority=5) Responding with outbound response: HTTP 200\\ncontent-length: 85\\ncontent-type: application/json\\n\\n{\"results\":true,\"internalTransactionId\":\"lB6-JwrDGgR-ZvKy\",\"executionTimeInMillis\":0}\n","container_name":"idm-geoservices","namespace_name":"dev","host":"server"}

To replace where I have a literal "\n", and present it as a New Line and also represent the JSON payload in the pretty JSON format, if possible.

I have tried something like:

| eval log=replace(log,"\\\\n","[\n]")

I can get the replacement to show when I do something like "TEST", but what I really want to do is have the "log" field present the \n in the message as a new line.

Longer term, we're going to implement Splunk Connect for Kubernetes, but we're trying to get our user taken care of with being able to parse out a multi-line JSON message from Kubernetes.

Thank you!
Stephen

Tags (3)
0 Karma
1 Solution

javiergn
Super Champion

Hi @skirven,

If you just want to present that JSON in a nicer format then your only option is probably the makemv command:

| makemv delim="\\\\n" yourfield

As in the following screenshot. I know is not ideal, as you are creating a multivalue field, but it is presented in a much more readable way so hopefully it'll help.

alt text

Or you could also use spath to extract the fields and then apply makemv to log:

alt text

Let me know if that helps.

Thanks,
J

View solution in original post

0 Karma

to4kawa
Ultra Champion
| makeresults 
| eval _raw="{\"log\":\"2020-05-28 06:52:34,671 GMT TRACE [com.xxx.oss.core.servlets.TransactionFilter] (http-nio-8080-exec-7|R:lB6-JwrDGgR-ZvKy|ThreadId=55|ThreadPriority=5) Responding with outbound response: HTTP 200\\ncontent-length: 85\\ncontent-type: application/json\\n\\n{\"results\":true,\"internalTransactionId\":\"lB6-JwrDGgR-ZvKy\",\"executionTimeInMillis\":0}\n\",\"container_name\":\"idm-geoservices\",\"namespace_name\":\"dev\",\"host\":\"server\"}"
| rex "(?ms)log\":\"(?<log>.*?\})"
| eval log=split(replace(log,"(?ms)\\\n","#"),"#")
| rex mode=sed "s/{.*?}[^,]+,(.*)/{\1/"
| spath
| eval _raw=mvindex(log,4)
| spath

That's a lot of work.

0 Karma

skirven
Communicator

Yes. That's a lot of work, and seems highly tailored to the one result. The problem I'm really just wanting to solve is to replace the "\n" with a normal regex \n and show the new line.

When the string was parsed with other data, it didn't represent the rest of the lines correctly. 😞
Thanks!
Stephen

0 Karma

to4kawa
Ultra Champion

| eval log=split(replace(log,"(?ms)\\\n","#"),"#")

If you want to solve the problem, see what the query is doing.
The respondent is not your subcontractor.

0 Karma

Patrick_Peeters
Splunk Employee
Splunk Employee

You can update the field with something like this:

<base search>| rex field=log mode=sed "s/\\\n/\n/g"

However that does not change the shown event, it only updates the field (visualise it with | table log for example).

To do it at index-time, make changes to props.conf like below:

[your_sourcetype]
SEDCMD-newline= s/\\\n/\n/g
0 Karma

skirven
Communicator

@ppeeters_splunk - Hi! I actually have tried that too, and it didn't work.

I think this also mirrors the test case, and it's not parsing out what I was hoping. 😞

| makeresults | eval _raw="{\"log\":\"2020-05-28 06:52:34,671 GMT TRACE [com.xxx.oss.core.servlets.TransactionFilter] (http-nio-8080-exec-7|R:lB6-JwrDGgR-ZvKy|ThreadId=55|ThreadPriority=5) Responding with outbound response: HTTP 200\\ncontent-length: 85\\ncontent-type: application/json\\n\\n{\"results\":true,\"internalTransactionId\":\"lB6-JwrDGgR-ZvKy\",\"executionTimeInMillis\":0}\n\",\"container_name\":\"idm-geoservices\",\"namespace_name\":\"dev\",\"host\":\"server\"}" | rex field=log mode=sed "s/\\\n/\n/g"
0 Karma

Patrick_Peeters
Splunk Employee
Splunk Employee

Ah ok, in my case I had already used sourcetype=_json when indexing the data. In that case you can use | rex field=_raw mode=sed "s/\\\n/\n/g" as there is no log field.

It won't make it pretty JSON format though, just show the event slightly better. The issue you'll have is that the log field is not proper JSON in the first place, it's just a long string. You might get it to work with some regex trickery or in a complex way but you might be better off seeing if you can get the source data properly formatted first.

0 Karma

javiergn
Super Champion

Hi @skirven,

If you just want to present that JSON in a nicer format then your only option is probably the makemv command:

| makemv delim="\\\\n" yourfield

As in the following screenshot. I know is not ideal, as you are creating a multivalue field, but it is presented in a much more readable way so hopefully it'll help.

alt text

Or you could also use spath to extract the fields and then apply makemv to log:

alt text

Let me know if that helps.

Thanks,
J

0 Karma

skirven
Communicator

@javiergn - That actually didn't get anything parsed? 😞

0 Karma

javiergn
Super Champion

Sorry what didn't get anything parsed?
This?

| makemv delim="\\\\n" yourfield

What's your fieldname? Is it _raw? If so then use it like:

| makemv delim="\\\\n" _raw

You can also extract our JSON fields first with spath, as in the second screenshot, and then use makemv against the one containing the new lines (log in this case).

In any case, it always helps if you paste query here.

0 Karma

skirven
Communicator

Looking at this solution closer, I actually proposed this solution to the user, and awaiting feedback. What I didn't think of was just dropping it into a table:

| makemv delim="\\n" log | table _time, log
0 Karma

skirven
Communicator

This worked! Thanks!

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...