Getting Data In

Modular Inputs - How can I split the HTTP JSON response into multiple events?

manuel2202
Explorer

How can I split the HTTP JSON response into multiple events?
My REST API is returning a JSON Array and for each array element I like to create separate event in Splunk.
I tried sys.stdout.flush() but without success. How can I set event boundaries?

This is my json response

[{"sever1": true}, {"sever2": true}]

This is my run method

def run_script():


    try:
        cfg = get_config()
        conn = httplib.HTTPConnection(cfg["url"])
        conn.request("GET", "/")
        r1 = conn.getresponse()
        data1 = json.loads(r1.read())

        for d in data1:
            print d
            sys.stdout.flush()

        conn.close()
    except Exception as e:
        logger.error(e)
    logger.info("RUN HTTP DONE")
Tags (3)
0 Karma

manuel2202
Explorer

Each array element should be a separate event.
I solved this problem. I overlooked the SCHEMA config.

It's important to set the streaming mode to xml

SCHEME = """<scheme>
    <title>Hello</title>
    <description>Log Data to Splunk</description>
    <streaming_mode>xml</streaming_mode>
</scheme>
"""

# Empty introspection routine
def do_scheme():
    print SCHEME

See Documentation:
https://docs.splunk.com/Documentation/Splunk/6.6.2/AdvancedDev/ModInputsStream

And than I can post separate events to Splunk by using this code.

# prints XML stream
def post_data(time, data):
    print "<stream><event unbroken=\"1\"><time>%s</time><data>%s</data><done/></event></stream>"  % (time,data)

It's also important to set unbroken attribute and and the tag.

See Documentation:
https://docs.splunk.com/Documentation/Splunk/6.6.2/AdvancedDev/ModInputsStream

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

I've moved your comment to an answer. Please accept your own answer here so that it will be flagged as having an accepted answer. - Thanks!

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

Is your JSON data actually all on one line that you want to separate into different events? That is, is it like this?:

[{"sever1": true}, {"sever2": true}]

or this?:

[{"sever1": true}, 
{"sever2": true}]

What is your purpose in separating the values into different events? This sort of thing can be done at search time without having to do it at index time.

0 Karma
Get Updates on the Splunk Community!

The Payment Operations Wake-Up Call: Why Financial Institutions Can't Afford ...

The same scenario plays out across financial institutions daily. A payment system fails at 11:30 AM on a busy ...

Make Your Case: A Ready-to-Send Letter for Getting Approval to Attend .conf25

Hello Splunkers, Want to attend .conf25 in Boston this year but not sure how to convince your manager? We've ...

Community Spotlight: A Splunk Expert's Journey

In the world of data analytics, some journeys leave a lasting impact not only on the individual but on the ...