All Apps and Add-ons

Is there any plans of development for incorporating MessageTraceDetail API into this App?

dpanych
Communicator

Hello, I've been using this app to collect Message Trace report logs, and while it works great, I'd like to know if there is any plans to include the MessageTraceDetail report API? This API pulls extra detailed information about each mail/message event. Here's a link to the API: https://msdn.microsoft.com/en-us/library/office/jj984328.aspx. To get that API to work, you need a MessageTraceId and RecipientAddress of an event (these are found in the MessageTrace API).

0 Karma
1 Solution

dpanych
Communicator

I was able to modify the script but found that it was too resource intensive... i.e. For every email that comes in, MessageTraceDetail would run on that message. It would run fine for a couple minutes but then it would hang. I figured a more efficient way to utilize MessageTraceDetail - build a ad-hoc Event Action button to run only on events when needed.

Here is the code I used to run MessageTraceDetail on every message:
FIle_name: input_module_ms_o365_message_trace.py
*Note: All logic is in the for loop of the file mentioned above. In the code block down below, I have commented out the logic for MessageTraceDetail.

            for message_trace in data["d"]["results"]:

            # According to https://msdn.microsoft.com/en-us/library/office/jj984335.aspx
            # The StartDate and EndDate fields do not provide useful information in the report results...
            message_trace.pop("StartDate")
            message_trace.pop("EndDate")

            if not index_metadata:
                message_trace.pop("__metadata")

            # Convert the /Date()/ format returned from the JSON and create a new field
            _received = re.search('/Date\((.+?)\)/', message_trace["Received"])
            if(_received):
                t = int(_received.group(1))

                # There is a chance that we could ingest duplicate data due to date granularity.
                # This check should catch those situations.
                if t <= max_epoch:
                    continue

                d = datetime.datetime.utcfromtimestamp(t/1000)
                message_trace["DateReceived"] = d.isoformat() + "Z"

                # Keep up with the max received date
                max_date = max([max_date, d])

                # Keep up with the max epoch as well for greater precision
                current_max_epoch = max([current_max_epoch,t])

            """url_detail = "https://reports.office365.com/ecp/ReportingWebService/Reporting.svc/MessageTraceDetail?$format=json&$filter=MessageTraceId eq guid'"+message_trace["MessageTraceId"]+"' and RecipientAddress eq '"+message_trace["RecipientAddress"]+"'"
            r_detail = requests.get(url_detail, auth=requests.auth.HTTPBasicAuth(global_microsoft_office_365_username, global_microsoft_office_365_password))

            data_detail = r_detail.json()

            for message_trace_detail in data_detail["d"]["results"]:
                message_trace_detail.pop("StartDate")
                message_trace_detail.pop("EndDate")

                _received_detail = re.search('/Date\((.+?)\)/', message_trace["Received"])
                if(_received_detail):
                    t_detail = int(_received_detail.group(1))

                d_detail = datetime.datetime.utcfromtimestamp(t_detail/1000)
                message_trace_detail["DateReceived"] = d_detail.isoformat() + "Z"
            """
            e = helper.new_event(source=helper.get_input_type(), index=helper.get_output_index(), sourcetype=helper.get_sourcetype(), data=json.dumps(message_trace))
            #e_detail = helper.new_event(source=helper.get_input_type(), index=helper.get_output_index(), sourcetype=helper.get_sourcetype(), data=json.dumps(message_trace_detail))

            ew.write_event(e)
            #ew.write_event(e_detail)

View solution in original post

0 Karma

dpanych
Communicator

I was able to modify the script but found that it was too resource intensive... i.e. For every email that comes in, MessageTraceDetail would run on that message. It would run fine for a couple minutes but then it would hang. I figured a more efficient way to utilize MessageTraceDetail - build a ad-hoc Event Action button to run only on events when needed.

Here is the code I used to run MessageTraceDetail on every message:
FIle_name: input_module_ms_o365_message_trace.py
*Note: All logic is in the for loop of the file mentioned above. In the code block down below, I have commented out the logic for MessageTraceDetail.

            for message_trace in data["d"]["results"]:

            # According to https://msdn.microsoft.com/en-us/library/office/jj984335.aspx
            # The StartDate and EndDate fields do not provide useful information in the report results...
            message_trace.pop("StartDate")
            message_trace.pop("EndDate")

            if not index_metadata:
                message_trace.pop("__metadata")

            # Convert the /Date()/ format returned from the JSON and create a new field
            _received = re.search('/Date\((.+?)\)/', message_trace["Received"])
            if(_received):
                t = int(_received.group(1))

                # There is a chance that we could ingest duplicate data due to date granularity.
                # This check should catch those situations.
                if t <= max_epoch:
                    continue

                d = datetime.datetime.utcfromtimestamp(t/1000)
                message_trace["DateReceived"] = d.isoformat() + "Z"

                # Keep up with the max received date
                max_date = max([max_date, d])

                # Keep up with the max epoch as well for greater precision
                current_max_epoch = max([current_max_epoch,t])

            """url_detail = "https://reports.office365.com/ecp/ReportingWebService/Reporting.svc/MessageTraceDetail?$format=json&$filter=MessageTraceId eq guid'"+message_trace["MessageTraceId"]+"' and RecipientAddress eq '"+message_trace["RecipientAddress"]+"'"
            r_detail = requests.get(url_detail, auth=requests.auth.HTTPBasicAuth(global_microsoft_office_365_username, global_microsoft_office_365_password))

            data_detail = r_detail.json()

            for message_trace_detail in data_detail["d"]["results"]:
                message_trace_detail.pop("StartDate")
                message_trace_detail.pop("EndDate")

                _received_detail = re.search('/Date\((.+?)\)/', message_trace["Received"])
                if(_received_detail):
                    t_detail = int(_received_detail.group(1))

                d_detail = datetime.datetime.utcfromtimestamp(t_detail/1000)
                message_trace_detail["DateReceived"] = d_detail.isoformat() + "Z"
            """
            e = helper.new_event(source=helper.get_input_type(), index=helper.get_output_index(), sourcetype=helper.get_sourcetype(), data=json.dumps(message_trace))
            #e_detail = helper.new_event(source=helper.get_input_type(), index=helper.get_output_index(), sourcetype=helper.get_sourcetype(), data=json.dumps(message_trace_detail))

            ew.write_event(e)
            #ew.write_event(e_detail)
0 Karma

claudius_satria
Loves-to-Learn

Hi dpanych,

 

may i know the version do you use? im using version 1.2.4

and can you explain more detail how to insert your script in input_module_ms_o365_message_trace.py ?

im using version 1.2.4

0 Karma
Get Updates on the Splunk Community!

SOC4Kafka - New Kafka Connector Powered by OpenTelemetry

The new SOC4Kafka connector, built on OpenTelemetry, enables the collection of Kafka messages and forwards ...

Your Voice Matters! Help Us Shape the New Splunk Lantern Experience

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Building Momentum: Splunk Developer Program at .conf25

At Splunk, developers are at the heart of innovation. That’s why this year at .conf25, we officially launched ...