All Apps and Add-ons

JSON how to break file into single events

dsofoulis
Path Finder

Hi Everyone.

I am using the API data input with Splunk to collect the following data. The format I'm using is JSON.
SAMPLE:

{
"Meta Data": {
"1. Information": "Daily Prices and Volumes for Digital Currency",
"2. Digital Currency Code": "BTC",
"3. Digital Currency Name": "Bitcoin",
"4. Market Code": "AUD",
"5. Market Name": "Australian Dollar",
"6. Last Refreshed": "2018-01-29 (end of day)",
"7. Time Zone": "UTC"
},
"Time Series (Digital Currency Daily)": {
"2018-01-29": {
"1a. open (AUD)": "14557.05214175",
"1b. open (USD)": "11804.12866653",
"2a. high (AUD)": "14582.48830689",
"2b. high (USD)": "11835.94535201",
"3a. low (AUD)": "13861.27196591",
"3b. low (USD)": "11216.74489015",
"4a. close (AUD)": "13919.77783987",
"4b. close (USD)": "11271.81445898",
"5. volume": "997.57467196",
"6. market cap (USD)": "11244476.61130983"
},
"2018-01-28": {
"1a. open (AUD)": "14229.70171702",
"1b. open (USD)": "11539.57330826",
"2a. high (AUD)": "14683.13628361",
"2b. high (USD)": "11907.28596490",
"3a. low (AUD)": "14202.27268193",
"3b. low (USD)": "11517.32973861",
"4a. close (AUD)": "14590.69388649",
"4b. close (USD)": "11831.40832999",
"5. volume": "874.38330435",
"6. market cap (USD)": "10345185.91069148"
},
"2018-01-27": {
"1a. open (AUD)": "13905.65975789",
"1b. open (USD)": "11276.79155663",
"2a. high (AUD)": "14362.62591110",
"2b. high (USD)": "11647.36815262",
"3a. low (AUD)": "13734.89429681",
"3b. low (USD)": "11138.30934556",
"4a. close (AUD)": "14229.60710190",
"4b. close (USD)": "11539.49658014",
"5. volume": "584.99890477",
"6. market cap (USD)": "6750592.86098091"
},
},

So far I've only been able to bring in the entire feed in one event. I would like to be able to break the feed into single events but cannot figure out how to achieve this.

Here is my Props.conf

[btc:json]
CHARSET = 
DATETIME_CONFIG = 
EVENT_BREAKER = .*\},
INDEXED_EXTRACTIONS = json
KV_MODE = none
MAX_TIMESTAMP_LOOKAHEAD = 550
TIME_FORMAT = %Y-%M-%d %H:%M:%S
TIME_PREFIX = \s+"
TRUNCATE = 500000
TZ = UTC
disabled = false
pulldown_type = true
0 Karma

Damien_Dallimor
Ultra Champion

Loads of answers for this already if you search (click on the App tag).

Such as : https://answers.splunk.com/answers/611916/help-with-custom-response-handler-for-rest-api-mod.html

0 Karma

dsofoulis
Path Finder

Thanks Damien for you response. I've had a look at the link and I'm using the TIME_PREFIX setting; TIME_PREFIX = \s+"

Looks like I need to drop the metadata header section i.e.

"Meta Data": {
"1. Information": "Daily Prices and Volumes for Digital Currency",
"2. Digital Currency Code": "BTC",
"3. Digital Currency Name": "Bitcoin",
"4. Market Code": "AUD",
"5. Market Name": "Australian Dollar",
"6. Last Refreshed": "2018-01-29 (end of day)",
"7. Time Zone": "UTC"
},

Also I need to break after },

Any ideas how to do this?

0 Karma

Damien_Dallimor
Ultra Champion

You need to add a custom response handler to rest_ta/bin/responsehandlers.py

This custom response handler will break up the raw json into individual events for you.

You then declare this custom response handler to be applied in your rest stanza setup.

Loads of examples in answers already, just search.

Maybe something like :

class ExampleHandler:

     def __init__(self,**args):
         pass

     def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
         if response_type == "json":        
             output = json.loads(raw_response_output)

             for item in output["Meta Data"]:
                 print_xml_stream(json.dumps(item))   
         else:
             print_xml_stream(raw_response_output)
0 Karma

dsofoulis
Path Finder

Hi Damien,

using the config which you given me I was able to get Splunk to index the following. (this is part of "Meta Data" heading and also it is the first part only, not key value pairs. I would like to index "Time Series (Digital Currency Daily)")

"4. Market Code"

"2. Digital Currency Code"

"1. Information"

"7. Last Refreshed"

"6. Interval"

"3. Digital Currency Name"

"5. Market Name"

"8. Time Zone"

"2. Digital Currency Code"

Been trying lots of different stuff including replace metadata with "Time Series (Digital Currency Daily)" and it stopped working completely.
Any idea what I'm missing?

0 Karma

dsofoulis
Path Finder

Oh ok, I've tried the code which you've given me but it did not work (i restarted splunkd to be sure the config had loaded). I'm not familiar with python. any chance you'd be able to knock up a config for me to put into my responsehandlers.py?

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) v3.54.0

The Splunk Threat Research Team (STRT) recently released Enterprise Security Content Update (ESCU) v3.54.0 and ...

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...