All Apps and Add-ons
Highlighted

JSON how to break file into single events

Path Finder

Hi Everyone.

I am using the API data input with Splunk to collect the following data. The format I'm using is JSON.
SAMPLE:

{
"Meta Data": {
"1. Information": "Daily Prices and Volumes for Digital Currency",
"2. Digital Currency Code": "BTC",
"3. Digital Currency Name": "Bitcoin",
"4. Market Code": "AUD",
"5. Market Name": "Australian Dollar",
"6. Last Refreshed": "2018-01-29 (end of day)",
"7. Time Zone": "UTC"
},
"Time Series (Digital Currency Daily)": {
"2018-01-29": {
"1a. open (AUD)": "14557.05214175",
"1b. open (USD)": "11804.12866653",
"2a. high (AUD)": "14582.48830689",
"2b. high (USD)": "11835.94535201",
"3a. low (AUD)": "13861.27196591",
"3b. low (USD)": "11216.74489015",
"4a. close (AUD)": "13919.77783987",
"4b. close (USD)": "11271.81445898",
"5. volume": "997.57467196",
"6. market cap (USD)": "11244476.61130983"
},
"2018-01-28": {
"1a. open (AUD)": "14229.70171702",
"1b. open (USD)": "11539.57330826",
"2a. high (AUD)": "14683.13628361",
"2b. high (USD)": "11907.28596490",
"3a. low (AUD)": "14202.27268193",
"3b. low (USD)": "11517.32973861",
"4a. close (AUD)": "14590.69388649",
"4b. close (USD)": "11831.40832999",
"5. volume": "874.38330435",
"6. market cap (USD)": "10345185.91069148"
},
"2018-01-27": {
"1a. open (AUD)": "13905.65975789",
"1b. open (USD)": "11276.79155663",
"2a. high (AUD)": "14362.62591110",
"2b. high (USD)": "11647.36815262",
"3a. low (AUD)": "13734.89429681",
"3b. low (USD)": "11138.30934556",
"4a. close (AUD)": "14229.60710190",
"4b. close (USD)": "11539.49658014",
"5. volume": "584.99890477",
"6. market cap (USD)": "6750592.86098091"
},
},

So far I've only been able to bring in the entire feed in one event. I would like to be able to break the feed into single events but cannot figure out how to achieve this.

Here is my Props.conf

[btc:json]
CHARSET = 
DATETIME_CONFIG = 
EVENT_BREAKER = .*\},
INDEXED_EXTRACTIONS = json
KV_MODE = none
MAX_TIMESTAMP_LOOKAHEAD = 550
TIME_FORMAT = %Y-%M-%d %H:%M:%S
TIME_PREFIX = \s+"
TRUNCATE = 500000
TZ = UTC
disabled = false
pulldown_type = true
0 Karma
Highlighted

Re: JSON how to break file into single events

Ultra Champion

Loads of answers for this already if you search (click on the App tag).

Such as : https://answers.splunk.com/answers/611916/help-with-custom-response-handler-for-rest-api-mod.html

0 Karma
Highlighted

Re: JSON how to break file into single events

Path Finder

Thanks Damien for you response. I've had a look at the link and I'm using the TIMEPREFIX setting; TIMEPREFIX = \s+"

Looks like I need to drop the metadata header section i.e.

"Meta Data": {
"1. Information": "Daily Prices and Volumes for Digital Currency",
"2. Digital Currency Code": "BTC",
"3. Digital Currency Name": "Bitcoin",
"4. Market Code": "AUD",
"5. Market Name": "Australian Dollar",
"6. Last Refreshed": "2018-01-29 (end of day)",
"7. Time Zone": "UTC"
},

Also I need to break after },

Any ideas how to do this?

0 Karma
Highlighted

Re: JSON how to break file into single events

Ultra Champion

You need to add a custom response handler to rest_ta/bin/responsehandlers.py

This custom response handler will break up the raw json into individual events for you.

You then declare this custom response handler to be applied in your rest stanza setup.

Loads of examples in answers already, just search.

Maybe something like :

class ExampleHandler:

     def __init__(self,**args):
         pass

     def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
         if response_type == "json":        
             output = json.loads(raw_response_output)

             for item in output["Meta Data"]:
                 print_xml_stream(json.dumps(item))   
         else:
             print_xml_stream(raw_response_output)
0 Karma
Highlighted

Re: JSON how to break file into single events

Path Finder

Oh ok, I've tried the code which you've given me but it did not work (i restarted splunkd to be sure the config had loaded). I'm not familiar with python. any chance you'd be able to knock up a config for me to put into my responsehandlers.py?

0 Karma
Highlighted

Re: JSON how to break file into single events

Path Finder

Hi Damien,

using the config which you given me I was able to get Splunk to index the following. (this is part of "Meta Data" heading and also it is the first part only, not key value pairs. I would like to index "Time Series (Digital Currency Daily)")

"4. Market Code"

"2. Digital Currency Code"

"1. Information"

"7. Last Refreshed"

"6. Interval"

"3. Digital Currency Name"

"5. Market Name"

"8. Time Zone"

"2. Digital Currency Code"

Been trying lots of different stuff including replace metadata with "Time Series (Digital Currency Daily)" and it stopped working completely.
Any idea what I'm missing?

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.