Monitor O365 IP/URL Updates with REST API Modular Input

[jodros]

I am currently experimenting monitoring Office365 IP/URL changes (https://support.content.office.net/en-us/static/O365IPAddresses.xml) with the REST API modular input. I am able to poll and index the O365 data as well as run spath against it. However, every line of XML is being line broken into individual events. I am not able to search on a parent grouping in the XML to output all values in that grouping. Below is an example. I am not able to search for product{@name}=o365 and it return all address contained.

<product name="o365">
<addresslist type="IPv6">
<address>2603:1030:800:5::bfee:a0ad</address>
<address>2620:1ec:34::/48</address>
<address>2620:1ec:38d::/48</address>
<address>2620:1ec:4::/48</address>
<address>2620:1ec:5::/48</address>
<address>2620:1ec:6::/48</address>
<address>2620:1ec:7::/48</address>
<address>2620:1ec:a::/48</address>
<address>2620:1ec:a92::/48</address>
<address>2620:1ec:b::/48</address>
<address>2620:1ec:c11::/48</address>
<address>2801:80:1d0:1c00::/64</address>
<address>2a01:111:2003::/48</address>
<address>2a01:111:200a:a::/64</address>
<address>2a01:111:202c::/48</address>
<address>2a01:111:202e::/48</address>
<address>2a01:111:202d::/48</address>
<address>2a01:111:2035:8::/64</address>
<address>2a01:111:f100:1004::4134:f0c8</address>
<address>2a01:111:f100:7000::6fdd:682b</address>
<address>2a01:111:f100:8001::d5c7:8077</address>
<address>2a01:111:f102:8001::1761:4237</address>
<address>2a01:111:f100:a001::a83f:5c85</address>
<address>2a01:111:f406:1::/64</address>
<address>2a01:111:f406:1000::/64</address>
<address>2a01:111:f406:1004::/64</address>
<address>2a01:111:f406:1801::/64</address>
<address>2a01:111:f406:1805::/64</address>
<address>2a01:111:f406:3404::/64</address>
<address>2A01:111:F406:8000::/64</address>
<address>2a01:111:f406:8801::/64</address>
<address>2a01:111:f406:a003::/64</address>
<address>2a01:111:f406:c00::/64</address>
<address>2001:489a:2101:100::/64</address>
</addresslist>

What I would like to do is monitor for any changes to the list and output an alert showing what has changed. Future state might be to modify devices on-prem via their respective API with Splunk based on this list changing.

I am also open to ideas on how to accomplish this in another way. Any input would be appreciated.

Thanks

[Damien_Dallimor]

I added a custom response handler definition in rest_ta/bin/responsehandlers.py :

class RemoveNewlinesHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):      
        print_xml_stream(raw_response_output.replace('\r\n','').replace('<?xml version="1.0" encoding="utf-8"?>',''))

Then wired this up in the setup for the REST input :

So everything gets indexed as 1 event , making searching easier :

See how you get on from here.

Then maybe look at the diff command for event comparisons :

index=main sourcetype=office | diff position1=1 position2=2

[jodros]

Thanks for the solution. I will try it and see how it works.

[Damien_Dallimor]

Does the raw XML response contain line feeds/carriage returns ?

[jodros]

Thanks for replying Damien. To view the raw XML response, click on the link provided above. I don't know if line breaks are included. However I have shifted my thinking. I don't think it matters so much how the data in being indexed. The goal is to compare the data set with a previous copy of the data set and identify any discrepancies, i.e. additions/deletions. I do not know the best way to accomplish this, possibly with lookup tables? Any insight would be great.

Thanks