Getting Data In

Format unicode characters in json input

vstariradev
Explorer

We're trying to index json formatted logs from kubernetes pods by removing the json formatting and making the logs appear like normal syslog input.

Using the below lines in props.conf:

SHOULD_LINEMERGE = false
SEDCMD-1_unjsonify = s/{"log":"(?:\\u[0-9]+)?(.*?)\\n","stream.*/\1/g
SEDCMD-2_unescapequotes = s/\\"/"/g
BREAK_ONLY_BEFORE={"logname":

We managed to transform the indexed logs from this:

{"log":"2019-07-18T14:11:48+00:00 kubernetes location1 0.0.0.0 - - [18/Jul/2019:14:11:48 +0000] \"GET /saml2/idp/sso?sp=something.com\u0026RelayState=https://something.something.com/URL/paths HTTP/1.1\" 200 2808 \"-\" \"jmeter\" \"something.something.com\" \"0.0.0.0\" 15982 \"95931E90-49DC-462D-B29F-86AF681A6B3B\"\n","stream":"stdout","time":"2019-07-18T14:11:48.485908193Z"}

to this:

2019-06-13T07:33:53+00:00 kubernetes location1 0.0.0.0
- - [13/Jun/2019:07:33:53 +0000] "POST /saml2/sp/acs/oac.something.com HTTP/1.1" 200 5573 "https://something.else.com/saml2/idp/sso?sp=something.com\u0026RelayState=https%3A%2F%2Fsomething.else.com%2Fadmin%2F" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:67.0) Gecko/20100101 Firefox/67.0" "something.something.com" "0.0.0.0" 9109 "A088E5DB-311C-400E-8AE9-A7B74CA7365C"

However, we get unconverted unicode characters like \u0026, %3A and %2F.

How can we convert them to normal unicode characters?

Thanks

0 Karma
Get Updates on the Splunk Community!

Security Professional: Sharpen Your Defenses with These .conf25 Sessions

Sooooooooooo, guess what. .conf25 is almost here, and if you're on the Security Learning Path, this is your ...

First Steps with Splunk SOAR

Our first step was to gather a list of the playbooks we wanted and to sort them by priority.  Once this list ...

How To Build a Self-Service Observability Practice with Splunk Observability Cloud

If you’ve read our previous post on self-service observability, you already know what it is and why it ...