JSON how to break file into single events

[dsofoulis]

Hi Everyone.

I am using the API data input with Splunk to collect the following data. The format I'm using is JSON.
SAMPLE:

{
"Meta Data": {
"1. Information": "Daily Prices and Volumes for Digital Currency",
"2. Digital Currency Code": "BTC",
"3. Digital Currency Name": "Bitcoin",
"4. Market Code": "AUD",
"5. Market Name": "Australian Dollar",
"6. Last Refreshed": "2018-01-29 (end of day)",
"7. Time Zone": "UTC"
},
"Time Series (Digital Currency Daily)": {
"2018-01-29": {
"1a. open (AUD)": "14557.05214175",
"1b. open (USD)": "11804.12866653",
"2a. high (AUD)": "14582.48830689",
"2b. high (USD)": "11835.94535201",
"3a. low (AUD)": "13861.27196591",
"3b. low (USD)": "11216.74489015",
"4a. close (AUD)": "13919.77783987",
"4b. close (USD)": "11271.81445898",
"5. volume": "997.57467196",
"6. market cap (USD)": "11244476.61130983"
},
"2018-01-28": {
"1a. open (AUD)": "14229.70171702",
"1b. open (USD)": "11539.57330826",
"2a. high (AUD)": "14683.13628361",
"2b. high (USD)": "11907.28596490",
"3a. low (AUD)": "14202.27268193",
"3b. low (USD)": "11517.32973861",
"4a. close (AUD)": "14590.69388649",
"4b. close (USD)": "11831.40832999",
"5. volume": "874.38330435",
"6. market cap (USD)": "10345185.91069148"
},
"2018-01-27": {
"1a. open (AUD)": "13905.65975789",
"1b. open (USD)": "11276.79155663",
"2a. high (AUD)": "14362.62591110",
"2b. high (USD)": "11647.36815262",
"3a. low (AUD)": "13734.89429681",
"3b. low (USD)": "11138.30934556",
"4a. close (AUD)": "14229.60710190",
"4b. close (USD)": "11539.49658014",
"5. volume": "584.99890477",
"6. market cap (USD)": "6750592.86098091"
},
},

So far I've only been able to bring in the entire feed in one event. I would like to be able to break the feed into single events but cannot figure out how to achieve this.

Here is my Props.conf

[btc:json]
CHARSET = 
DATETIME_CONFIG = 
EVENT_BREAKER = .*\},
INDEXED_EXTRACTIONS = json
KV_MODE = none
MAX_TIMESTAMP_LOOKAHEAD = 550
TIME_FORMAT = %Y-%M-%d %H:%M:%S
TIME_PREFIX = \s+"
TRUNCATE = 500000
TZ = UTC
disabled = false
pulldown_type = true

[Damien_Dallimor]

Loads of answers for this already if you search (click on the App tag).

Such as : https://answers.splunk.com/answers/611916/help-with-custom-response-handler-for-rest-api-mod.html

[dsofoulis]

Thanks Damien for you response. I've had a look at the link and I'm using the TIME_PREFIX setting; TIME_PREFIX = \s+"

Looks like I need to drop the metadata header section i.e.

"Meta Data": {
"1. Information": "Daily Prices and Volumes for Digital Currency",
"2. Digital Currency Code": "BTC",
"3. Digital Currency Name": "Bitcoin",
"4. Market Code": "AUD",
"5. Market Name": "Australian Dollar",
"6. Last Refreshed": "2018-01-29 (end of day)",
"7. Time Zone": "UTC"
},

Also I need to break after },

Any ideas how to do this?

[Damien_Dallimor]

You need to add a custom response handler to rest_ta/bin/responsehandlers.py

This custom response handler will break up the raw json into individual events for you.

You then declare this custom response handler to be applied in your rest stanza setup.

Loads of examples in answers already, just search.

Maybe something like :

class ExampleHandler:

     def __init__(self,**args):
         pass

     def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
         if response_type == "json":        
             output = json.loads(raw_response_output)

             for item in output["Meta Data"]:
                 print_xml_stream(json.dumps(item))   
         else:
             print_xml_stream(raw_response_output)

[dsofoulis]

Hi Damien,

using the config which you given me I was able to get Splunk to index the following. (this is part of "Meta Data" heading and also it is the first part only, not key value pairs. I would like to index "Time Series (Digital Currency Daily)")

"4. Market Code"

"2. Digital Currency Code"

"1. Information"

"7. Last Refreshed"

"6. Interval"

"3. Digital Currency Name"

"5. Market Name"

"8. Time Zone"

"2. Digital Currency Code"

Been trying lots of different stuff including replace metadata with "Time Series (Digital Currency Daily)" and it stopped working completely.
Any idea what I'm missing?

[dsofoulis]

Oh ok, I've tried the code which you've given me but it did not work (i restarted splunkd to be sure the config had loaded). I'm not familiar with python. any chance you'd be able to knock up a config for me to put into my responsehandlers.py?