All Apps and Add-ons

REST API Modular Input: How to split out modular REST input into multiple events?

Explorer

I have a modular call made every 5 minutes to a performance checking API that validates all of our microservices are performing ok. Currently in our lab its onboarded as a single one line event.

{"https://web-call-one.co.uk":{"TimeStarted":"2017-03-21T14:57:26.638684208Z","TimeStartedUnix":1490108246,"TimeToLastByte":19,"ContentLength":15928,"TimeToFirstByte":19,"EndpointURL":"https://web-call-one.co.uk","SpeedInKB":1690.118726383912,"TimeToConnect":1,"HTTPStatusCode":200},"https://web-call-two.co.uk":{"TimeStarted":"2017-03-21T14:57:52.981022715Z","TimeStartedUnix":1490108272,"TimeToLastByte":3,"ContentLength":1184637,"TimeToFirstByte":1,"EndpointURL":"https://web-call-two.co.uk","SpeedInKB":49103.969662105934,"TimeToConnect":0,"HTTPStatusCode":200},"https://web-call-three.co.uk":{"TimeStarted":"2017-03-21T14:57:26.022008698Z","TimeStartedUnix":1490108246,"TimeToLastByte":19,"ContentLength":127135,"TimeToFirstByte":15,"EndpointURL":"https://web-call-three.co.uk","SpeedInKB":3224.4794945448634,"TimeToConnect":1,"HTTPStatusCode":200}} 

if i change the format in an editor these are what the distinct events look like....

    "https://web-call-one.co.uk": {
        "TimeStarted": "2017-03-21T14:57:26.638684208Z",
        "TimeStartedUnix": 1490108246,
        "TimeToLastByte": 19,
        "ContentLength": 15928,
        "TimeToFirstByte": 19,
        "EndpointURL": "https://web-call-one.co.uk",
        "SpeedInKB": 1690.118726383912,
        "TimeToConnect": 1,
        "HTTPStatusCode": 200
    },
    "https://web-call-two.co.uk": {
        "TimeStarted": "2017-03-21T14:57:52.981022715Z",
        "TimeStartedUnix": 1490108272,
        "TimeToLastByte": 3,
        "ContentLength": 1184637,
        "TimeToFirstByte": 1,
        "EndpointURL": "https://web-call-two.co.uk",
        "SpeedInKB": 49103.969662105934,
        "TimeToConnect": 0,
        "HTTPStatusCode": 200
    },
    "https://web-call-three.co.uk": {
        "TimeStarted": "2017-03-21T14:57:26.022008698Z",
        "TimeStartedUnix": 1490108246,
        "TimeToLastByte": 19,
        "ContentLength": 127135,
        "TimeToFirstByte": 15,
        "EndpointURL": "https://web-call-three.co.uk",
        "SpeedInKB": 3224.4794945448634,
        "TimeToConnect": 1,
        "HTTPStatusCode": 200
    },

ideally i'd like to have these be ingested as separate events.... LINE_BREAKER won't play ball and i've tried a few of the custom handlers but can't quite get them worked out. this is what i tried to no avail at present...

class MyRestHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
        if response_type == "json":
            output = json.loads(raw_response_output)
            for event in output["TimeStarted"]:
                print_xml_stream(json.dumps(event))
        else:
            print_xml_stream(raw_response_output)

Any assistance would be great

0 Karma

Esteemed Legend

Try this:

LINE_BREAKER = (^{|},|}})(?:"https|\w*$)
SHOULD_LINEMERGE = false
0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!