Hi,
I am having difficulty parsing out some raw JSON data. Each day Splunk is required to hit an API and pull back the previous days data. Splunk can connect and pull the data back without any issues, it's just the parsing causing me headaches.
A sample of the raw data is below. There are thousands of events for each day in the extract, two events are in the sample.
{"source":"ABC-trade-tracker","identifierType":"ABC","identifier":"2015-01-12","propertyData":{"SDPABCSALES2_20150112_001":{"clearingCode":"12345678","creationAction":"CustRequestedQuote","instrumentDescription":"Product 23","instrumentId":"123456","marketer":"","notional":"10000000","productType":"ABC","settlementDate":"20150112","side":"RECEIVE","state":"Done","tradeDate":"2015-01-12","tradeId":"SDPABCSALES2_20150112_001","type":"RFQ","updateTimeStamp":"2015-01-12 09:03:48","user":"tester2","userData":"Tester 31","userText":"TRADX - ANY","value":"1.7800000000000002"},"XABCA_A3o_0":{"instrumentDescription":"product 38","instrumentId":"12131654","killTime":"","marketer":"","markets":"US, UK, AUS","notional":"55000000","notionalFill":"0","productType":"ABC","side":"Sell","state":"Error","timeInForce":"FAS","tradeDate":"2015-01-12","tradeId":"XABCA_A3o_0","type":"Limit order","updateTimeStamp":"2015-01-12 23:10:20","user":"tester3","userRole":"client","value":"0.78"}},"_links":{"self":{"href":"https://api-test.test.net/ABC/2015-01-12"}}}
About as close as I have got is configuring the props.conf and transforms.conf below. (Which I know will not get the desired result but its the closest I've got)
Props.conf
[pockdbapi2]
SHOULD_LINEMERGE = false
LINE_BREAKER = (\},)
REPORT-all = pockdbapi3tr
TRUNCATE = 0
MAX_EVENTS = 500000
TIME_PREFIX = ("updateTimeStamp":)
TIME_FORMAT = %Y-%m-%d %H:%M:%S
transforms.conf
[pockdbapi3tr]
DELIMS = ",", ":"
Which results in the below raw events.
The events are all split correctly apart from the first event. The timestamp is correct for each new event and all fields extract correctly.
{"source":"ABC-trade-tracker","identifierType":"ABC","identifier":"2015-01-12","propertyData":{"SDPABCSALES2_20150112_001":{"clearingCode":"12345678","creationAction":"CustRequestedQuote","instrumentDescription":"Product 23","instrumentId":"123456","marketer":"","notional":"10000000","productType":"ABC","settlementDate":"20150112","side":"RECEIVE","state":"Done","tradeDate":"2015-01-12","tradeId":"SDPABCSALES2_20150112_001","type":"RFQ","updateTimeStamp":"2015-01-12 09:03:48","user":"tester2","userData":"Tester 31","userText":"TRADX - ANY","value":"1.7800000000000002"
"XABCA_A3o_0":{"instrumentDescription":"product 38","instrumentId":"12131654","killTime":"","marketer":"","markets":"US, UK, AUS","notional":"55000000","notionalFill":"0","productType":"ABC","side":"Sell","state":"Error","timeInForce":"FAS","tradeDate":"2015-01-12","tradeId":"XABCA_A3o_0","type":"Limit order","updateTimeStamp":"2015-01-12 23:10:20","user":"tester3","userRole":"client","value":"0.78"}
"_links":{"self":{"href":"https://api-test.test.net/ABC/2015-01-12"}}}
All events apart from the first event parse as I would like.
Please can anyone advise how I would split out the below from the first event?
{"source":"ABC-trade-tracker","identifierType":"ABC","identifier":"2015-01-12","propertyData":{
Do I need to move away from LINE_BREAKER to get the desired result?
Thanks,
Dan
... View more