Getting Data In
Highlighted

Modular Inputs - How can I split the HTTP JSON response into multiple events?

Explorer

How can I split the HTTP JSON response into multiple events?
My REST API is returning a JSON Array and for each array element I like to create separate event in Splunk.
I tried sys.stdout.flush() but without success. How can I set event boundaries?

This is my json response

[{"sever1": true}, {"sever2": true}]

This is my run method

def run_script():


    try:
        cfg = get_config()
        conn = httplib.HTTPConnection(cfg["url"])
        conn.request("GET", "/")
        r1 = conn.getresponse()
        data1 = json.loads(r1.read())

        for d in data1:
            print d
            sys.stdout.flush()

        conn.close()
    except Exception as e:
        logger.error(e)
    logger.info("RUN HTTP DONE")
Tags (3)
0 Karma
Highlighted

Re: Modular Inputs - How can I split the HTTP JSON response into multiple events?

SplunkTrust
SplunkTrust

Is your JSON data actually all on one line that you want to separate into different events? That is, is it like this?:

[{"sever1": true}, {"sever2": true}]

or this?:

[{"sever1": true}, 
{"sever2": true}]

What is your purpose in separating the values into different events? This sort of thing can be done at search time without having to do it at index time.

0 Karma
Highlighted

Re: Modular Inputs - How can I split the HTTP JSON response into multiple events?

Explorer

Each array element should be a separate event.
I solved this problem. I overlooked the SCHEMA config.

It's important to set the streaming mode to xml

SCHEME = """<scheme>
    <title>Hello</title>
    <description>Log Data to Splunk</description>
    <streaming_mode>xml</streaming_mode>
</scheme>
"""

# Empty introspection routine
def do_scheme():
    print SCHEME

See Documentation:
https://docs.splunk.com/Documentation/Splunk/6.6.2/AdvancedDev/ModInputsStream

And than I can post separate events to Splunk by using this code.

# prints XML stream
def post_data(time, data):
    print "<stream><event unbroken=\"1\"><time>%s</time><data>%s</data><done/></event></stream>"  % (time,data)

It's also important to set unbroken attribute and and the tag.

See Documentation:
https://docs.splunk.com/Documentation/Splunk/6.6.2/AdvancedDev/ModInputsStream

0 Karma
Highlighted

Re: Modular Inputs - How can I split the HTTP JSON response into multiple events?

SplunkTrust
SplunkTrust

I've moved your comment to an answer. Please accept your own answer here so that it will be flagged as having an accepted answer. - Thanks!

0 Karma