topic Re: How is linebreaking of events configured when receiving XML logs via REST input? in All Apps and Add-ons

How is linebreaking of events configured when receiving XML logs via REST input?

woodcock — Wed, 10 Jun 2015 20:10:21 GMT

I am new to REST inputs and am having a very basic problem that I do not see discussed before. I am using http GET and xml for Response type. I am receiving the response just fine but only as 1 giant event. I would like to split events at the "<alarm>" tag (ending at "</alarm>") but I do not see any way to configure this. On a lark, I went ahead and put a props.conf file in the same app directory as this input with the normal settings that would work for a file-based input but they have no effect. How is linbreaking of events controlled for REST inputs?

Re: How is linebreaking of events configured when receiving XML logs via REST input?

bmacias84 — Wed, 10 Jun 2015 22:27:49 GMT

This is lightly documented in the documentation in the section Custom Response Handlers. The response handlers can be found in the bin directory in a file call responsehandlers.py. By the Modular input use DefaultResponseHandler which just pass the REST response to STDOUT. You can change it to any of the ones find the file by added the following setting to the stanza of your input, response_handler= <class>. class is just one of the classes from responsehandlers.py.

From the sounds of it your REST response gives you multiple events in the message which you need to split or else it comes in as one giant event. Looking at responsehandlers.py it looks like you will have to write a little bit of python which should take about 15 lines of code.

Not know how your raw data looks I might write something like this. This was not tested as just wrote of the hip.

class AlarmResponseHandler:

  def __init__(self,**args):
    pass

  def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
    #for each line in response.  Assuming it reads like a text file with new line char returns
    for line in raw_response_output:
      # if ',/alarm>' in the line end of event
      if '</alarm>' in line:
        event = event += line
        print_xml_stream(event)
        # clear event for new event
        event = None

Re: How is linebreaking of events configured when receiving XML logs via REST input?

Damien_Dallimor — Thu, 11 Jun 2015 00:40:50 GMT

Can you post an example of the full XML you receive. I can then help you with a sample custom response handler.

Re: How is linebreaking of events configured when receiving XML logs via REST input?

ncrisler — Thu, 11 Jun 2015 02:27:16 GMT

<alarm-list>
<alarm>
<assignedBy/>
<assignedTo/>
<custom1/>
<custom2/>
<custom3/>
<custom4/>
<custom5/>
<devId>D344459B39C33B452C53464089C59E78B</devId>
<domain>Place</domain>
<hostname>name</hostname>
<hub>name</hub>
<id>ZC40864030-00086</id>
<level>2</level>
<message>
ssss.exe: Process ssss.exe is not running
</message>
<metId>MF66466D228CC0462DB58C5CA12D2C789</metId>
<nas>servername</nas>
<origin>servername</origin>
<prevLevel>0</prevLevel>
<probe>processes</probe>
<robot>server</robot>
<severity>Warning</severity>
<source>111.111.111.1111</source>
<subsystem>Process</subsystem>
<subsystemId>1.1.2</subsystemId>
<suppressionCount>0</suppressionCount>
<suppressionKey>process/ssss.exe [1]/process_state</suppressionKey>
<timeArrival>2015-06-10T03:59:15-04:00</timeArrival>
<timeOrigin>2015-06-10T03:59:04-04:00</timeOrigin>
<timeReceived>2015-06-10T03:59:15-04:00</timeReceived>
<userTag1>customer</userTag1>
<userTag2>Production</userTag2>
<visible>true</visible>
</alarm>
<alarm>
<assignedBy/>
<assignedTo/>
<custom1/>
<custom2/>
<custom3/>
<custom4/>
<custom5/>
<devId>D1D431F7A35BB0912BB52FDF175FD3307</devId>
<domain>org/domain>
<hostname>serverName</hostname>
<hub>ServerHub</hub>
<id>YN44419525-32263</id>
<level>4</level>
<message>TrackerSvr PMQ: 219 files, expected < 50</message>
<metId>MF580138FE25DA78DF96521183CA3245A</metId>
<nas>IND1109C2NM13_NMS</nas>
<origin>mel063gnh021</origin>
<prevLevel>0</prevLevel>
<probe>dirscan</probe>
<robot>serverName</robot>
<severity>Major</severity>
<source>22.22.22.22</source>
<subsystem>Probe</subsystem>
<subsystemId>1.2.3</subsystemId>
<suppressionCount>1431</suppressionCount>
<suppressionKey>dirscan/TrackerSvr PMQ/number_of_files</suppressionKey>
<timeArrival>2015-06-08T05:13:29-04:00</timeArrival>
<timeOrigin>2015-02-09T12:05:17-05:00</timeOrigin>
<timeReceived>2015-06-08T05:13:29-04:00</timeReceived>
<userTag1>Customer Name</userTag1>
<userTag2>Production</userTag2>
<visible>false</visible>
</alarm>
<alarm-list>

Re: How is linebreaking of events configured when receiving XML logs via REST input?

Damien_Dallimor — Thu, 11 Jun 2015 03:10:29 GMT

Try something like :

class AlarmHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
        if response_type == "xml": 
            import xml.etree.ElementTree as ET
            alarm_list = ET.fromstring(encodeXMLText(raw_response_output))
            for alarm in alarm_list:
                alarm_xml_str = ET.tostring(alarm, encoding='utf8', method='xml')
                print_xml_stream(alarm_xml_str)               

        else:
            print_xml_stream(raw_response_output)

Re: How is linebreaking of events configured when receiving XML logs via REST input?

ncrisler — Thu, 11 Jun 2015 03:24:53 GMT

Should I just be adding this to the bottom of responsehandlers.py ?

Re: How is linebreaking of events configured when receiving XML logs via REST input?

Damien_Dallimor — Thu, 11 Jun 2015 03:34:16 GMT

Yes.you will also see other examples in that file.note , I just gave an example only, may or may not need a little tweaking.

Re: How is linebreaking of events configured when receiving XML logs via REST input?

ncrisler — Thu, 11 Jun 2015 03:36:21 GMT

Ok just checking. I am trying it now.

Re: How is linebreaking of events configured when receiving XML logs via REST input?

ncrisler — Wed, 01 Jul 2015 18:15:04 GMT

For some reason when I added AlarmHandler to the bottom of responsehandlers it broke the polling of the REST data. Do I need to add import xml to the top of the config?

Re: How is linebreaking of events configured when receiving XML logs via REST input?

bmacias84 — Wed, 01 Jul 2015 18:45:48 GMT

Shouldn't have to as it imported when the class is called on line 8.

Re: How is linebreaking of events configured when receiving XML logs via REST input?

ncrisler — Wed, 01 Jul 2015 19:22:20 GMT

Does this look right?

#add your custom response handler class to this module
import json
import datetime



#the default handler , does nothing , just passes the raw output directly to STDOUT
class DefaultResponseHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
        cookies = response_object.cookies
        if cookies:
            req_args["cookies"] = cookies        
        print_xml_stream(raw_response_output)

#template
class MyResponseHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):        
        print_xml_stream("foobar")

'''various example handlers follow'''

class BoxEventHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
        if response_type == "json":        
            output = json.loads(raw_response_output)
            if not "params" in req_args:
                req_args["params"] = {}
            if "next_stream_position" in output:    
                req_args["params"]["stream_position"] = output["next_stream_position"]
            for entry in output["entries"]:
                print_xml_stream(json.dumps(entry))   
        else:
            print_xml_stream(raw_response_output)  

class QualysGuardActivityLog:
    '''Response handler for QualysGuard activity log.'''

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
        if not "params" in req_args:
            req_args["params"] = {}
        date_from = (datetime.datetime.now() - datetime.timedelta(minutes=1)).strftime("%Y-%m-%dT%H:%M:%SZ")
        req_args["params"]["date_from"] = date_from
        print_xml_stream(raw_response_output) 

class FourSquareCheckinsEventHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
        if response_type == "json":        
            output = json.loads(raw_response_output)
            last_created_at = 0
            for checkin in output["response"]["checkins"]["items"]:
                print_xml_stream(json.dumps(checkin)) 
                if "createdAt" in checkin:
                    created_at = checkin["createdAt"]
                    if created_at > last_created_at:
                        last_created_at = created_at
            if not "params" in req_args:
                req_args["params"] = {}

            req_args["params"]["afterTimestamp"] = last_created_at

        else:
            print_xml_stream(raw_response_output) 

class ThingWorxTagHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
        if response_type == "json":        
            output = json.loads(raw_response_output)
            for row in output["rows"]:
                print_xml_stream(json.dumps(row))                      
        else:
            print_xml_stream(raw_response_output) 

class FireEyeEventHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
        if response_type == "json":        
            output = json.loads(response_object.content)
            last_display_id = -1
            for alert in output["alerts"]:
                print_xml_stream(json.dumps(alert))  
                if "displayId" in alert:
                    display_id = alert["displayId"]
                    if display_id > last_display_id:
                        last_display_id = display_id
            if not "params" in req_args:
                req_args["params"] = {}

            if last_display_id > -1:
                req_args["params"]["offset"] = last_display_id

        else:
            print_xml_stream(raw_response_output) 


class BugsenseErrorsEventHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
        if response_type == "json":        
            output = json.loads(raw_response_output)

            for error in output["data"]:
                print_xml_stream(json.dumps(error))   
        else:
            print_xml_stream(raw_response_output)

class MyCustomHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):

        req_args["data"] = 'What does the fox say'   

        print_xml_stream(raw_response_output)


class TwitterEventHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):       

        if response_type == "json":        
            output = json.loads(raw_response_output)
            last_tweet_indexed_id = 0
            for twitter_event in output["statuses"]:
                print_xml_stream(json.dumps(twitter_event))
                if "id_str" in twitter_event:
                    tweet_id = twitter_event["id_str"]
                    if tweet_id > last_tweet_indexed_id:
                        last_tweet_indexed_id = tweet_id

            if not "params" in req_args:
                req_args["params"] = {}

            req_args["params"]["since_id"] = last_tweet_indexed_id

        else:
            print_xml_stream(raw_response_output)

class JSONArrayHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
        if response_type == "json":
            output = json.loads(raw_response_output)

            for entry in output:
                print_xml_stream(json.dumps(entry))
        else:
            print_xml_stream(raw_response_output)


#HELPER FUNCTIONS

# prints XML stream
def print_xml_stream(s):
    print "<stream><event unbroken=\"1\"><data>%s</data><done/></event></stream>" % encodeXMLText(s)



def encodeXMLText(text):
    text = text.replace("&", "&amp;")
    text = text.replace("\"", "&quot;")
    text = text.replace("'", "&apos;")
    text = text.replace("<", "&lt;")
    text = text.replace(">", "&gt;")
    text = text.replace("\n", "")
    return text

class AlarmHandler:

     def __init__(self,**args):
         pass

     def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
         if response_type == "xml": 
             import xml.etree.ElementTree as ET
             alarm_list = ET.fromstring(encodeXMLText(raw_response_output))
             for alarm in alarm_list:
                 alarm_xml_str = ET.tostring(alarm, encoding='utf8', method='xml')
                 print_xml_stream(alarm_xml_str)               

         else:
             print_xml_stream(raw_response_output)