topic Re: How can I strip the headers for JSON to parse correctly? Currently not working! in Splunk Search

How can I strip the headers for JSON to parse correctly? Currently not working!

danfein — Thu, 29 Dec 2016 21:23:56 GMT

I am trying to remove the header of my JSON data but my current setup will not work, it continues to parse with the header (and therefore doesn't actually parse!).

The header / beginning of the JSON looks like this:

2016-12-29T14:52:53-05:00 17.1.9.10 {"creationTime": 1483993708000

I tried this:

[sourcename]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
CHARSET=UTF-8
INDEXED_EXTRACTIONS=json
category=Structured
disabled=false
pulldown_type=true
TZ=America/New_York
SEDCMD-StripHeader = s/^[^{]+//
KV_MODE = json
EXTRACT-_time={"creationTime":
TIME_PREFIX=\{\"creationTime\"\:
TZ=America/New_York
MAX_TIMESTAMP_LOOKAHEAD=13

Any idea why this is not working?

Many thanks in advance!

Re: How can I strip the headers for JSON to parse correctly? Currently not working!

koshyk — Fri, 30 Dec 2016 04:45:37 GMT

I had similar issue and was solved using answers
https://answers.splunk.com/answers/117121/extract-json-data-within-the-logs.html

I would split the logic into two parts. (1) To extract whole JSON out (2) To extract key value pairs within JSON
In props.conf

### props.conf
 [myjson]
 REPORT-json = report-json,report-json-kv

In transforms.conf

[report-json]       
 # This will get the json payload from the logs. 
 REGEX = (?P<json1>{\"creationTime\".+)

  [report-json-kv]
  SOURCE_KEY = json1
  KV_MODE = json
  NO_BINARY_CHECK = 1
  TRUNCATE = 0
  TIME_PREFIX=\{\"creationTime\"\:
  TZ=America/New_York
  SHOULD_LINEMERGE = true
  BREAK_ONLY_BEFORE = someregularexpression
  MAX_TIMESTAMP_LOOKAHEAD=13

Re: How can I strip the headers for JSON to parse correctly? Currently not working!

danfein — Tue, 29 Sep 2020 12:14:06 GMT

Unfortunately this didn't work for me, if I am adding data in manually to test, would I be able to see this in the UI or only at search time?

Additionally, for the BREAK_ONLY_BEFORE, do I want everything leading up the the {\"creationTime or everything after?

Thanks!

Re: How can I strip the headers for JSON to parse correctly? Currently not working!

koshyk — Tue, 29 Sep 2020 12:14:14 GMT

The above settings are REPORT. So search time. You can change to "TRANSFORMS-json" to make it index time.
break_only_before is something you need to try out and test as I don't have the full json. normally it is }\n

Re: How can I strip the headers for JSON to parse correctly? Currently not working!

danfein — Fri, 30 Dec 2016 18:27:54 GMT

Ok I will try to make this work; it may be unorthodox but do you have Skype or teamviewer? This is really driving me in circles!

Re: How can I strip the headers for JSON to parse correctly? Currently not working!

danfein — Sat, 31 Dec 2016 20:34:12 GMT

I really don't understand how this works/breaks down. Can you describe the approach, please?