Splunk Search

How can I extract nested JSON at index time as their own event?

DHastie
Engager

I am using the REST API to get a large sample of JSON data every minute from the Bittrex Exchange but I would like to get each of the nested objects as their own event.
A sample of the data.

{
"success": true,
"message": "",
"result": [
{
"MarketName": "BTC-1ST",
"High": 0.00004356,
"Low": 0.00003995,
"Volume": 275838.36163835,
"Last": 0.00004,
"BaseVolume": 11.26409631,
"TimeStamp": "2018-02-22T14:26:44.74",
"Bid": 0.0000399,
"Ask": 0.0000404,
"OpenBuyOrders": 189,
"OpenSellOrders": 4095,
"PrevDay": 0.00004236,
"Created": "2017-06-06T01:22:35.727"
},
{
"MarketName": "BTC-2GIVE",
"High": 0.00000118,
"Low": 0.00000113,
"Volume": 845591.96670095,
"Last": 0.00000114,
"BaseVolume": 0.96678559,
"TimeStamp": "2018-02-22T14:26:43.647",
"Bid": 0.00000114,
"Ask": 0.00000115,
"OpenBuyOrders": 195,
"OpenSellOrders": 1235,
"PrevDay": 0.00000118,
"Created": "2016-05-16T06:44:15.287"
},...
]
}

I also have a props.conf

[bittrex-json]
TRANSFORMS-nullJsonNestingStart= removeNestingStart
TRANSFORMS-nullJsonNestingEnd= removeNestingEnd
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = true
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%f
category = Application
disabled = false
pulldown_type = true

and a the transforms.conf

[removeNestingStart]
REGEX = (\{\"success\":true,\"message\":\"\",\"result\":\[)
DEST_KEY = queue
FORMAT = nullQueue

[removeNestingEnd]
REGEX = (\]\})
DEST_KEY=queue
FORMAT = nullQueue

So I tested the regex and it matches only the top level before nested objects start. It also only catches the final "] }".
If my understanding is correct, this set up should keep the nested portions to be indexed as the other sections are sent to nullQueue.
However, when I use my props.conf sourcetype the events preview shows 0 events. Somehow my regex is matching and sending all the data to Nullqueue.

I know that I can use the spath command to extract the objects at search time. However as each object has it's own timestamp within, I'd like each object to be it's own event.
Any help is appreciated.

0 Karma

DHastie
Engager

Ok, in the question I am trying to remove the top level Json and trailing ]} so what remains is a bunch of Json objects {...} separated by commas.
The regex strings matched and worked but the nullQueue options will remove the whole event where it finds a match to the regex. So I was unintentionally removing the whole event.
What I needed to use was the SEDCMD in props.conf like so

SEDCMD-removeNestingStart = s/(\{\"success\":true,\"message\":\"\",\"result\":\[)//
SEDCMD-removeNestingEnd =s/(\]\})//

This succesfully removes only the matched string.

Hopefully this'll help any one elses mis-understanding

0 Karma
Get Updates on the Splunk Community!

Video | Welcome Back to Smartness, Pedro

Remember Splunk Community member, Pedro Borges? If you tuned into Episode 2 of our Smartness interview series, ...

Detector Best Practices: Static Thresholds

Introduction In observability monitoring, static thresholds are used to monitor fixed, known values within ...

Expert Tips from Splunk Education, Observability in Action, Plus More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...