Getting Data In

Modular Inputs - How can I split the HTTP JSON response into multiple events?

manuel2202
Explorer

How can I split the HTTP JSON response into multiple events?
My REST API is returning a JSON Array and for each array element I like to create separate event in Splunk.
I tried sys.stdout.flush() but without success. How can I set event boundaries?

This is my json response

[{"sever1": true}, {"sever2": true}]

This is my run method

def run_script():


    try:
        cfg = get_config()
        conn = httplib.HTTPConnection(cfg["url"])
        conn.request("GET", "/")
        r1 = conn.getresponse()
        data1 = json.loads(r1.read())

        for d in data1:
            print d
            sys.stdout.flush()

        conn.close()
    except Exception as e:
        logger.error(e)
    logger.info("RUN HTTP DONE")
Tags (3)
0 Karma

manuel2202
Explorer

Each array element should be a separate event.
I solved this problem. I overlooked the SCHEMA config.

It's important to set the streaming mode to xml

SCHEME = """<scheme>
    <title>Hello</title>
    <description>Log Data to Splunk</description>
    <streaming_mode>xml</streaming_mode>
</scheme>
"""

# Empty introspection routine
def do_scheme():
    print SCHEME

See Documentation:
https://docs.splunk.com/Documentation/Splunk/6.6.2/AdvancedDev/ModInputsStream

And than I can post separate events to Splunk by using this code.

# prints XML stream
def post_data(time, data):
    print "<stream><event unbroken=\"1\"><time>%s</time><data>%s</data><done/></event></stream>"  % (time,data)

It's also important to set unbroken attribute and and the tag.

See Documentation:
https://docs.splunk.com/Documentation/Splunk/6.6.2/AdvancedDev/ModInputsStream

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

I've moved your comment to an answer. Please accept your own answer here so that it will be flagged as having an accepted answer. - Thanks!

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

Is your JSON data actually all on one line that you want to separate into different events? That is, is it like this?:

[{"sever1": true}, {"sever2": true}]

or this?:

[{"sever1": true}, 
{"sever2": true}]

What is your purpose in separating the values into different events? This sort of thing can be done at search time without having to do it at index time.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...