How can I extract nested JSON at index time as their own event?

DHastie — Thu, 22 Feb 2018 16:28:27 GMT

I am using the REST API to get a large sample of JSON data every minute from the Bittrex Exchange but I would like to get each of the nested objects as their own event.
A sample of the data.

{ "success": true, "message": "", "result": [ { "MarketName": "BTC-1ST", "High": 0.00004356, "Low": 0.00003995, "Volume": 275838.36163835, "Last": 0.00004, "BaseVolume": 11.26409631, "TimeStamp": "2018-02-22T14:26:44.74", "Bid": 0.0000399, "Ask": 0.0000404, "OpenBuyOrders": 189, "OpenSellOrders": 4095, "PrevDay": 0.00004236, "Created": "2017-06-06T01:22:35.727" }, { "MarketName": "BTC-2GIVE", "High": 0.00000118, "Low": 0.00000113, "Volume": 845591.96670095, "Last": 0.00000114, "BaseVolume": 0.96678559, "TimeStamp": "2018-02-22T14:26:43.647", "Bid": 0.00000114, "Ask": 0.00000115, "OpenBuyOrders": 195, "OpenSellOrders": 1235, "PrevDay": 0.00000118, "Created": "2016-05-16T06:44:15.287" },... ] }

I also have a props.conf

[bittrex-json]
TRANSFORMS-nullJsonNestingStart= removeNestingStart
TRANSFORMS-nullJsonNestingEnd= removeNestingEnd
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = true
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%f
category = Application
disabled = false
pulldown_type = true

and a the transforms.conf

[removeNestingStart]
REGEX = (\{\"success\":true,\"message\":\"\",\"result\":\[)
DEST_KEY = queue
FORMAT = nullQueue

[removeNestingEnd]
REGEX = (\]\})
DEST_KEY=queue
FORMAT = nullQueue

So I tested the regex and it matches only the top level before nested objects start. It also only catches the final "] }".
If my understanding is correct, this set up should keep the nested portions to be indexed as the other sections are sent to nullQueue.
However, when I use my props.conf sourcetype the events preview shows 0 events. Somehow my regex is matching and sending all the data to Nullqueue.

I know that I can use the spath command to extract the objects at search time. However as each object has it's own timestamp within, I'd like each object to be it's own event.
Any help is appreciated.

Re: How can I extract nested JSON at index time as their own event?

DHastie — Fri, 23 Feb 2018 09:42:32 GMT

Ok, in the question I am trying to remove the top level Json and trailing ]} so what remains is a bunch of Json objects {...} separated by commas.
The regex strings matched and worked but the nullQueue options will remove the whole event where it finds a match to the regex. So I was unintentionally removing the whole event.
What I needed to use was the SEDCMD in props.conf like so

SEDCMD-removeNestingStart = s/(\{\"success\":true,\"message\":\"\",\"result\":\[)//
SEDCMD-removeNestingEnd =s/(\]\})//

This succesfully removes only the matched string.

Hopefully this'll help any one elses mis-understanding

topic Re: How can I extract nested JSON at index time as their own event? in Splunk Search

How can I extract nested JSON at index time as their own event?

Re: How can I extract nested JSON at index time as their own event?