Hello, I've been using this app to collect Message Trace report logs, and while it works great, I'd like to know if there is any plans to include the MessageTraceDetail report API? This API pulls extra detailed information about each mail/message event. Here's a link to the API: https://msdn.microsoft.com/en-us/library/office/jj984328.aspx. To get that API to work, you need a MessageTraceId and RecipientAddress of an event (these are found in the MessageTrace API).
I was able to modify the script but found that it was too resource intensive... i.e. For every email that comes in, MessageTraceDetail would run on that message. It would run fine for a couple minutes but then it would hang. I figured a more efficient way to utilize MessageTraceDetail - build a ad-hoc Event Action button to run only on events when needed.
Here is the code I used to run MessageTraceDetail on every message:
FIle_name: input_module_ms_o365_message_trace.py
*Note: All logic is in the for loop of the file mentioned above. In the code block down below, I have commented out the logic for MessageTraceDetail.
for message_trace in data["d"]["results"]:
# According to https://msdn.microsoft.com/en-us/library/office/jj984335.aspx
# The StartDate and EndDate fields do not provide useful information in the report results...
message_trace.pop("StartDate")
message_trace.pop("EndDate")
if not index_metadata:
message_trace.pop("__metadata")
# Convert the /Date()/ format returned from the JSON and create a new field
_received = re.search('/Date\((.+?)\)/', message_trace["Received"])
if(_received):
t = int(_received.group(1))
# There is a chance that we could ingest duplicate data due to date granularity.
# This check should catch those situations.
if t <= max_epoch:
continue
d = datetime.datetime.utcfromtimestamp(t/1000)
message_trace["DateReceived"] = d.isoformat() + "Z"
# Keep up with the max received date
max_date = max([max_date, d])
# Keep up with the max epoch as well for greater precision
current_max_epoch = max([current_max_epoch,t])
"""url_detail = "https://reports.office365.com/ecp/ReportingWebService/Reporting.svc/MessageTraceDetail?$format=json&$filter=MessageTraceId eq guid'"+message_trace["MessageTraceId"]+"' and RecipientAddress eq '"+message_trace["RecipientAddress"]+"'"
r_detail = requests.get(url_detail, auth=requests.auth.HTTPBasicAuth(global_microsoft_office_365_username, global_microsoft_office_365_password))
data_detail = r_detail.json()
for message_trace_detail in data_detail["d"]["results"]:
message_trace_detail.pop("StartDate")
message_trace_detail.pop("EndDate")
_received_detail = re.search('/Date\((.+?)\)/', message_trace["Received"])
if(_received_detail):
t_detail = int(_received_detail.group(1))
d_detail = datetime.datetime.utcfromtimestamp(t_detail/1000)
message_trace_detail["DateReceived"] = d_detail.isoformat() + "Z"
"""
e = helper.new_event(source=helper.get_input_type(), index=helper.get_output_index(), sourcetype=helper.get_sourcetype(), data=json.dumps(message_trace))
#e_detail = helper.new_event(source=helper.get_input_type(), index=helper.get_output_index(), sourcetype=helper.get_sourcetype(), data=json.dumps(message_trace_detail))
ew.write_event(e)
#ew.write_event(e_detail)
I was able to modify the script but found that it was too resource intensive... i.e. For every email that comes in, MessageTraceDetail would run on that message. It would run fine for a couple minutes but then it would hang. I figured a more efficient way to utilize MessageTraceDetail - build a ad-hoc Event Action button to run only on events when needed.
Here is the code I used to run MessageTraceDetail on every message:
FIle_name: input_module_ms_o365_message_trace.py
*Note: All logic is in the for loop of the file mentioned above. In the code block down below, I have commented out the logic for MessageTraceDetail.
for message_trace in data["d"]["results"]:
# According to https://msdn.microsoft.com/en-us/library/office/jj984335.aspx
# The StartDate and EndDate fields do not provide useful information in the report results...
message_trace.pop("StartDate")
message_trace.pop("EndDate")
if not index_metadata:
message_trace.pop("__metadata")
# Convert the /Date()/ format returned from the JSON and create a new field
_received = re.search('/Date\((.+?)\)/', message_trace["Received"])
if(_received):
t = int(_received.group(1))
# There is a chance that we could ingest duplicate data due to date granularity.
# This check should catch those situations.
if t <= max_epoch:
continue
d = datetime.datetime.utcfromtimestamp(t/1000)
message_trace["DateReceived"] = d.isoformat() + "Z"
# Keep up with the max received date
max_date = max([max_date, d])
# Keep up with the max epoch as well for greater precision
current_max_epoch = max([current_max_epoch,t])
"""url_detail = "https://reports.office365.com/ecp/ReportingWebService/Reporting.svc/MessageTraceDetail?$format=json&$filter=MessageTraceId eq guid'"+message_trace["MessageTraceId"]+"' and RecipientAddress eq '"+message_trace["RecipientAddress"]+"'"
r_detail = requests.get(url_detail, auth=requests.auth.HTTPBasicAuth(global_microsoft_office_365_username, global_microsoft_office_365_password))
data_detail = r_detail.json()
for message_trace_detail in data_detail["d"]["results"]:
message_trace_detail.pop("StartDate")
message_trace_detail.pop("EndDate")
_received_detail = re.search('/Date\((.+?)\)/', message_trace["Received"])
if(_received_detail):
t_detail = int(_received_detail.group(1))
d_detail = datetime.datetime.utcfromtimestamp(t_detail/1000)
message_trace_detail["DateReceived"] = d_detail.isoformat() + "Z"
"""
e = helper.new_event(source=helper.get_input_type(), index=helper.get_output_index(), sourcetype=helper.get_sourcetype(), data=json.dumps(message_trace))
#e_detail = helper.new_event(source=helper.get_input_type(), index=helper.get_output_index(), sourcetype=helper.get_sourcetype(), data=json.dumps(message_trace_detail))
ew.write_event(e)
#ew.write_event(e_detail)
Hi dpanych,
may i know the version do you use? im using version 1.2.4
and can you explain more detail how to insert your script in input_module_ms_o365_message_trace.py ?
im using version 1.2.4