ITSI - Exchange Content pack - Some empty panels - Total_bytes not parsed correctly

[corti77]

Hi,

I run Splunk 9.0.3 with IT Essentials 4.15.0 with Exchange content pack 1.5.1 (DA-ITSI-CP-microsoft-exchange). We have an Exchange 2016 deployment on-premises. 

Reviewing the built-in dashboards,  I saw empty panels in some dashboards. An example is the panel "Outbound Message Volume" in the "Outbound Messages - Microsoft Exchange" dashboard. (see attachment) 

I dug into the query and replaced all macros, the resulting query was:

eventtype=smtp-outbound 
| join message_id 
    [ search eventtype=storedriver-receive 
    | fields message_id,sender] 
| eval sender=lower(sender) 
| eval sender_domain=lower(sender_domain) 
| eval sender_username=lower(sender_username) 
| eval recipients=lower(recipients)|eval recipient=lower(recipient)|eval recipient_domain=lower(recipient_domain)|eval recipient_username=lower(recipient_username)
| table _time,message_id,sc_ip,sender,recipient_count,recipients,total_bytes 
| eval total_kb=total_bytes/1024 
| timechart fixedrange=t bins=120 per_second(total_kb) as "Bandwidth"

the chart is created based on the value of total_kb which is calculated based on the extracted field total_bytes. I removed the last command (timechart) and total_bytes does not exist, so total_kb is not calculated.

I tried to find the issue and the eventtype corresponds to the sourcetype MSExchange:2013:MessageTracking  . I looked into the props.conf in the path <drive>:\Program Files\Splunk\etc\apps\DA-ITSI-CP-microsoft-exchange\default  and there are no evals created for the total_bytes field. 

[MSExchange:2013:MessageTracking]
SHOULD_LINEMERGE = false
CHECK_FOR_HEADER = false
REPORT-fields = msexchange2013msgtrack-fields,msgtrack-extract-psender,msgtrack-psender,msgtrack-sender,msgtrack-recipients,msgtrack-recipient
TRANSFORMS-comments = ignore_comments
FIELDALIAS-server_hostname_as_dest = server_hostname AS dest
FIELDALIAS-host_as_dvc = host AS dvc
EVAL-src=coalesce(original_client_ip,cs_ip)
EVAL-product = "Exchange"
EVAL-vendor = "Microsoft"
LOOKUP-event_id_to_action = event_id_to_action_lookup event_id OUTPUT action
FIELDALIAS-user = sender_username AS user
FIELDALIAS-orig_dest = ss_ip AS orig_dest
FIELDALIAS-dest_ip = ss_ip AS dest_ip
FIELDALIAS-return_addr = return_path AS return_addr
FIELDALIAS-size = message_size AS size
FIELDALIAS-subject = message_subject AS subject
EVAL-orig_src=coalesce(original_client_ip,cs_ip)
EVAL-protocol = "SMTP"
EVAL-vendor_product = "Microsoft Exchange"
EVAL-sender = coalesce(PurportedSender,sender)
EVAL-src_user = coalesce(PurportedSender,sender)
EVAL-sender_username = coalesce(psender_username,sender_username)
EVAL-sender_domain = coalesce(psender_domain,sender_domain)

I also checked msexchange2013msgtrack-fields entry in the transforms.conf and the field "total_bytes" appears there.

[msexchange2013msgtrack-fields]
FIELDS = "date_time","cs_ip","client_hostname","ss_ip","server_hostname","source_context","connector_id","source_id","event_id","internal_message_id","message_id","network_message_id","recipients","recipient_status","total_bytes","recipient_count","related_recipient_address","reference","message_subject","sender","return_path","message_info","directionality","tenant_id","original_client_ip","original_server_ip","custom_data"
DELIMS = ,

As a final check, I look for the Exchange logs and the total_bytes field is included in the logs. 
In the extract below the total_bytes appears in the correct position with a value of 55115.

#Software: Microsoft Exchange Server
#Version: 15.01.2507.021
#Log-type: Message Tracking Log
#Date: 2023-05-18T09:00:00.691Z
#Fields: date-time,client-ip,client-hostname,server-ip,server-hostname,source-context,connector-id,source,event-id,internal-message-id,message-id,network-message-id,recipient-address,recipient-status,total-bytes,recipient-count,related-recipient-address,reference,message-subject,sender-address,return-path,message-info,directionality,tenant-id,original-client-ip,original-server-ip,custom-data,transport-traffic-type,log-id,schema-version
2023-05-18T09:00:00.691Z,,HOST.xxx.y.z,,HOST,08DB13554C308059;2023-05-18T09:00:00.637Z;ClientSubmitTime:2023-05-18T09:00:00.117Z,,STOREDRIVER,DELIVER,140492675219457,<b133b1b22b184c049dacea930775bae5@xxx.yyy.z>,bfb98eea-8ce0-42a6-a016-08db577e42ed,mary@xx.y,,55115,1,,,RE: ArcGISDataDevTraining,john@xx.y,john@xx.y,2023-05-18T09:00:00.120Z;SRV=XXX.yy.z:TOTAL-SUB=0.234|SA=0.194|MTSS-PEN=0.041(MTSSD-PEN=0.037(MTSSDA=0.002|MTSSDC=0.008|SDSSO-PEN=0.012 (SMSC=0.008(X-SMSDR=0.001)|MTSSDM-PEN=0.004)));SRV=XXX.yyy.zz:TOTAL-HUB=270.010|SMR=0.145(SMRDI=0.006|SMRC=0.138(SMRCL=0.107|X-SMRCR=0.138))|CAT=0.124(CATORES=0.016 (CATRS=0.016(CATRS-Transport Rule Agent=0.004(X-ETREX=0.004)|CATRS-Index Routing Agent=0.011 ))|CATORT=0.104(CATRT=0.104(CATRT-Journal Agent=0.104)))|QDM=0.010;SRV=ATLHQMPHSMX1.eusc.europa.eu:TOTAL-DEL=0.060|SMR=0.006(SMRDI=0.005)|SDD=0.053(SDDSPCR=0.003(SDDCC=0.003)|SDDSPCS=0.002(SDDOS=0.002)|SDDPM=0.019(SDDPM-Conversations Processing Agent=0.012|SDDPM-Mailbox Rules Agent=0.004)|SDDSCMG=0.007(SDDCMM=0.002)|SDDCM=0.001|SDDSDMG=0.017(SDDR=0.017)|X-SDDS=0.011),Originating,,192.168.X.X,192.168.X.X,"S:IncludeInSla=True;S:MailboxDatabaseGuid=d3cbc250-34d2-4a36-8f6e-dab3d1248894;S:Mailboxes=ce6cae16-5bd9-4b7d-a1c4-9ae851224466;S:StoreObjectIds=AAAAAFjvGJqmWmRHocS0e5d51KAHAHn/uaFmuTFBgJ5aRTROcxAABSlL0VcAANwUEnI+lypImRLmR1/oEQoAA2WHe9sAAA==;S:FromEntity=Hosted;S:ToEntity=Hosted;S:P2RecipStat=0,003/1;S:MsgRecipCount=1;S:SubRecipCount=1;S:DeliveryLatency=0.571;S:AttachCount=1;S:E2ELatency=0.572;S:DeliveryPriority=Normal;S:AccountForest=xxx.yyyy.x",Email,a0cd35de-8a46-490d-ec84-08db577e4322,15.01.2507.021

what could be the reason why it does not get parsed correctly?

Cheers