Splunk Search

Separate multiple same name Multivalue fields

Path Finder

We are trying to alert on O365 service messages data. Under the "Messages" multivalue field, we are trying to pull the most recent multivalue field based on PublishedTime. We want to separate each multivalue field and report the most recent.

Search string:

index="o365data" sourcetype="o365:service:message" Id=EX212047 
| stats count by Messages{}.PublishedTime, WorkloadDisplayName, Messages{}.MessageText, Id 
| rename WorkloadDisplayName AS Workload Id AS Ticket Messages{}.MessageText AS Messages 
| fields - count
| tail 1

Example output:
alt text

Raw Event example:

{"ActionType": null, "AdditionalDetails": [{"Name": "NotifyInApp", "Value": "True"}], "AffectedTenantCount": 0, "AffectedUserCount": null, "AffectedWorkloadDisplayNames": [], "AffectedWorkloadNames": [], "Classification": "Incident", "EndTime": null, "Feature": "Access", "FeatureDisplayName": "E-Mail and calendar access", "Id": "EX212047", "ImpactDescription": "Users may receive repeated credential prompts within the Outlook client.", "LastUpdatedTime": "2020-05-08T16:59:41.103Z", "MessageType": "Incident", "Messages": [{"MessageText": "Title: Multiple credential prompts in the Outlook client\n\nUser Impact: Users may receive repeated credential prompts within the Outlook client.\n\nCurrent status: We're investigating a potential issue with multiple credential prompts. We'll provide an update within 30 minutes.", "PublishedTime": "2020-05-05T14:29:21.21Z"}, {"MessageText": "Title: Multiple credential prompts in the Outlook client\n\nUser Impact: Users may receive repeated credential prompts within the Outlook client.\n\nCurrent status: We're analyzing system logs to determine the source of the issue.\n\nScope of impact: This issue may affect any user that has service-based search or Focused Inbox.\n\nNext update by: Tuesday, May 5, 2020, at 5:00 PM\u00a0UTC", "PublishedTime": "2020-05-05T15:04:35.827Z"}, {"MessageText": "Title: Multiple credential prompts in the Outlook client\n\nUser Impact: Some users may receive repeated credential prompts within the Outlook client.\n\nMore info: Some users have reported that after multiple credential prompts they are able to access the service.\n\nCurrent status: Our investigation into the system logs did not provide enough data to determine the source of the issue. We're contacting affected users to gather Fiddler network trace logs, Support and Recovery Assistant logs, and examples of users that can reproduce the issue so that we can understand the root cause and create a strategy to remediate impact.\n\nScope of impact: This issue may affect any user that has service-based search or Focused Inbox.\n\nNext update by: Wednesday, May 6, 2020, at 6:30 PM\u00a0UTC", "PublishedTime": "2020-05-05T16:39:57.187Z"}, {"MessageText": "Title: Multiple credential prompts in the Outlook client\n\nUser Impact: Users may receive repeated credential prompts within the Outlook client.\n\nMore info: As a workaround, customer who are able to use Modern Authentication may enable it to mitigate impact for affected users. Details on how to enable Modern Authentication can be found here: https://docs.microsoft.com/en-us/exchange/clients-and-mobile-in-exchange-online/enable-or-disable-mo... status: We're analyzing trace logs provided by affected users to isolate the origin of the issue and determine our next steps. We've received some reports that enabling Modern Authentication mitigates the problem, though we're investigating how this relates to the cause of the problem.\n\nScope of impact: This issue may affect any user that has service-based search or Focused Inbox.\n\nNext update by: Wednesday, May 6, 2020, at 8:30 PM\u00a0UTC", "PublishedTime": "2020-05-06T17:45:17.867Z"}, {"MessageText": "Title: Multiple credential prompts in the Outlook client\n\nUser Impact: Users may receive repeated credential prompts within the Outlook client.\n\nMore info: As a workaround, customer who are able to use Modern Authentication may enable it to mitigate impact for affected users. Details on how to enable Modern Authentication can be found here: https://docs.microsoft.com/en-us/exchange/clients-and-mobile-in-exchange-online/enable-or-disable-mo... status: We're continuing to investigate the Fiddler network traces and data supplied by impacted users to isolate the cause.\n\nScope of impact: This issue may affect any user that has service-based search or Focused Inbox.\n\nNext update by: Wednesday, May 6, 2020, at 10:30 PM\u00a0UTC", "PublishedTime": "2020-05-06T20:34:02.627Z"}, {"MessageText": "Title: Multiple credential prompts in the Outlook client\n\nUser Impact: Users may receive repeated credential prompts within the Outlook client.\n\nMore info: While we're focused on remediation, users that have access to other protocols such as Outlook on the web or mobile devices can access their email without issue. Additionally, customers who are able to use Modern Authentication may enable it to mitigate impact for affected users. Details on how to enable Modern Authentication can be found here: https://docs.microsoft.com/en-us/exchange/clients-and-mobile-in-exchange-online/enable-or-disable-mo... status: We've created test accounts using various configurations to help us understand the exact circumstances that causes this issue to better define the scope of the problem and how we may go about resolving the impact.\n\nScope of impact: This issue may affect any user that has service-based search or Focused Inbox.\n\nNext update by: Thursday, May 7, 2020, at 1:30 AM\u00a0UTC", "PublishedTime": "2020-05-06T22:29:05.717Z"}, {"MessageText": "Title: Multiple credential prompts in the Outlook client\n\nUser impact: Users may receive repeated credential prompts within the Outlook client.\n\nMore info: While we're focused on remediation, users that have access to other protocols such as Outlook on the web or mobile devices can access their email without issue. Additionally, customers who are able to use Modern Authentication may enable it to mitigate impact for affected users. Details on how to enable Modern Authentication can be found here: https://docs.microsoft.com/en-us/exchange/clients-and-mobile-in-exchange-online/enable-or-disable-mo... status: We've determined that a recent Exchange Online update contains a code issue which is resulting in repeated credential prompts. We've halted deployment of the build to prevent further spread of impact and we're discussing mitigation steps for this event. We've confirmed that this issue appears to only affect basic authentication configuration users.\n\nScope of impact: This issue may affect any user that has service-based search or Focused Inbox. Additionally, this issue only affects users that are attempting to connect via basic authentication.\n\nRoot cause: A recent update to the Exchange Online service contains a code issue that is causing repeated credential prompts for basic authentication users.\n\nNext update by: Thursday, May 7, 2020, at 3:30 AM UTC", "PublishedTime": "2020-05-07T00:33:22.087Z"}, {"MessageText": "Title: Multiple credential prompts in the Outlook client\n\nUser Impact: Users may receive repeated credential prompts within the Outlook client.\n\nMore info: While we're focused on remediation, users that have access to other protocols such as Outlook on the web or mobile devices can access their email without issue. Additionally, customers who are able to use Modern Authentication may enable it to mitigate impact for affected users. Details on how to enable Modern Authentication can be found here: https://docs.microsoft.com/en-us/exchange/clients-and-mobile-in-exchange-online/enable-or-disable-mo... status: We've developed a fix and we're performing extensive validation to confirm that it'll resolve the issue. We expect to begin deployment of the fix within the next 14 hours.\n\nScope of impact: This issue may affect any user that has service-based search or Focused Inbox. Additionally, this issue only affects users that are attempting to connect to their Outlook Desktop clients via basic authentication. \n\nStart time: Tuesday, May 5, 2020, at 4:00 AM\u00a0UTC\n\nRoot cause: A recent update to the Exchange Online service contains a code issue that is causing repeated credential prompts for basic authentication users.\n\nNext update by: Thursday, May 7, 2020, at 5:00 PM\u00a0UTC", "PublishedTime": "2020-05-07T02:33:29.743Z"}, {"MessageText": "Title: Multiple credential prompts in the Outlook client\n\nUser Impact: Users may receive repeated credential prompts within the Outlook client.\n\nMore info: While we're focused on remediation, users that have access to other protocols such as Outlook on the web or mobile devices can access their email without issue.\n\nThis issue only impacts customers using Basic Authentication. Further, customers who are able to use Modern Authentication may enable it to mitigate impact for affected users; however, this process can require several hours to take effect for some customers.\n\nDetails on how to enable Modern Authentication can be found here: https://docs.microsoft.com/en-us/exchange/clients-and-mobile-in-exchange-online/enable-or-disable-mo... status: We're continuing to perform validation of our fix and expect this process to complete within the next three to four hours. Once complete, we\u2019ll initiate deployment of our solution to the affected infrastructure.\n\nScope of impact: This issue affects a subset of customers and users who are connecting to the service using basic authentication and utilize service-based search or a Focused inbox.\n\nStart time: Tuesday, May 5, 2020, at 4:00 AM UTC\n\nRoot cause: A recent update to the Exchange Online service contains a code issue that is causing repeated credential prompts for basic authentication users.\n\nNext update by: Thursday, May 7, 2020, at 11:00 PM UTC", "PublishedTime": "2020-05-07T16:54:26.367Z"}, {"MessageText": "Title: Multiple credential prompts in the Outlook client\n\nUser Impact: Users may receive repeated credential prompts within the Outlook client.\n\nMore info: While we're focused on remediation, users that have access to other protocols such as Outlook on the web or mobile devices can access their email without issue.\n\nThis issue only impacts customers using basic authentication. Customers who are able to use Modern Authentication may enable it to mitigate impact for affected users; however, this process can require several hours to take effect for some customers.\n\nWhile we understand this may not be a viable workaround for all customers we're committed to identifying and provided all potential solutions. Details on how to enable Modern Authentication can be found here: https://docs.microsoft.com/en-us/exchange/clients-and-mobile-in-exchange-online/enable-or-disable-mo... status: We've initiated deployment of our solution and users should experience service restoration as the fix reaches their environment. We will provide an estimated resolution timeline as soon as one is available.\n\nScope of impact: This issue affects a subset of customers and users who are connecting to the service using Basic Authentication and utilize service-based search or a Focused inbox.\n\nStart time: Tuesday, May 5, 2020, at 4:00 AM UTC\n\nPreliminary root cause: A recent update to the Exchange Online service contains a code issue that is causing repeated credential prompts for basic authentication users.\n\nNext update by: Friday, May 8, 2020, at 3:00 AM UTC", "PublishedTime": "2020-05-07T20:41:29.763Z"}, {"MessageText": "Title: Multiple credential prompts in the Outlook client\n\nUser Impact: Users may receive repeated credential prompts within the Outlook client.\n\nMore info: While we're focused on remediation, users that have access to other protocols such as Outlook on the web or mobile devices can access their email without issue.\n\nThis issue only impacts customers using basic authentication. Customers who are able to use Modern Authentication may enable it to mitigate impact for affected users; however, this process can require several hours to take effect for some customers.\n\nWhile we understand this may not be a viable workaround for all customers we're committed to identifying and providing all potential solutions. Details on how to enable Modern Authentication can be found here: https://docs.microsoft.com/en-us/exchange/clients-and-mobile-in-exchange-online/enable-or-disable-mo... status: We're continuing deployment of the fix and expect it reach all of the affected environments within the next 72 hours.\n\nScope of impact: This issue affects a subset of customers and users who are connecting to the service using Basic Authentication and utilize service-based search or a Focused inbox.\n\nStart time: Tuesday, May 5, 2020, at 4:00 AM UTC\n\nPreliminary root cause: A recent update to the Exchange Online service contains a code issue that is causing repeated credential prompts for basic authentication users.\n\nNext update by: Friday, May 8, 2020, at 5:00 PM UTC", "PublishedTime": "2020-05-08T00:13:31.147Z"}, {"MessageText": "Title: Multiple credential prompts in the Outlook client\n\nUser Impact: Users may receive repeated credential prompts within the Outlook client.\n\nMore info: While we're focused on remediation, users that have access to other protocols such as Outlook on the web or mobile devices can access their email without issue.\n\nThis issue only impacts customers using basic authentication. Customers who are able to use Modern Authentication may enable it to mitigate impact for affected users; however, this process can require several hours to take effect for some customers.\n\nWhile we understand this may not be a viable workaround for all customers we're committed to identifying and providing all potential solutions. Details on how to enable Modern Authentication can be found here: https://docs.microsoft.com/en-us/exchange/clients-and-mobile-in-exchange-online/enable-or-disable-mo... some customers who have disabled Modern Authentication, there is a secondary issue that is causing the client to attempt to use Modern Authentication regardless of the setting. Once the fix has been deployed, the client and service should use the expected configuration.\n\nCurrent status: We're closely monitoring progress of the fix deployment which has reached approximately 33 percent of the affected infrastructure. We expect that the deployment will complete within the next 48 hours.\n\nScope of impact: This issue affects a subset of customers and users who are connecting to the service using Basic Authentication and utilize service-based search or a Focused inbox.\n\nStart time: Tuesday, May 5, 2020, at 4:00 AM UTC\n\nEstimated time to resolve: Based on current progress, we expect deployment of the solution to complete by 11:00 PM UTC on Sunday, May 10, 2020. Customers should experience incremental service restoration as the deployment progresses.\n\nPreliminary root cause: A recent update to the Exchange Online service contains a code issue that is causing repeated credential prompts for basic authentication users.\n\nNext update by: Saturday, May 9, 2020, at 11:00 PM UTC", "PublishedTime": "2020-05-08T16:59:41.103Z"}], "PostIncidentDocumentUrl": null, "Severity": "Sev2", "StartTime": "2020-05-05T04:00:00Z", "Status": "Restoring service", "Title": "Multiple credential prompts in the Outlook client", "UserFunctionalImpact": "", "Workload": "Exchange", "WorkloadDisplayName": "Exchange Online"}

Syntax Highlighted Event Example:

{ [-]
   ActionType: null
   AdditionalDetails: [ [+]
   ]
   AffectedTenantCount: 0
   AffectedUserCount: null
   AffectedWorkloadDisplayNames: [ [+]
   ]
   AffectedWorkloadNames: [ [+]
   ]
   Classification: Incident
   EndTime: null
   Feature: Access
   FeatureDisplayName: E-Mail and calendar access
   Id: EX212047
   ImpactDescription: Users may receive repeated credential prompts within the Outlook client.
   LastUpdatedTime: 2020-05-08T16:59:41.103Z
   MessageType: Incident
   Messages: [ [-]
     { [-]
       MessageText: Title: Multiple credential prompts in the Outlook client

User Impact: Users may receive repeated credential prompts within the Outlook client.

Current status: We're investigating a potential issue with multiple credential prompts. We'll provide an update within 30 minutes.
       PublishedTime: 2020-05-05T14:29:21.21Z
     }
     { [+]
     }
     { [+]
     }
     { [+]
     }
     { [+]
     }
     { [+]
     }
     { [+]
     }
     { [+]
     }
     { [+]
     }
     { [+]
     }
     { [+]
     }
     { [-]
       MessageText: Title: Multiple credential prompts in the Outlook client

User Impact: Users may receive repeated credential prompts within the Outlook client.

More info: While we're focused on remediation, users that have access to other protocols such as Outlook on the web or mobile devices can access their email without issue.

This issue only impacts customers using basic authentication. Customers who are able to use Modern Authentication may enable it to mitigate impact for affected users; however, this process can require several hours to take effect for some customers.

While we understand this may not be a viable workaround for all customers we're committed to identifying and providing all potential solutions. Details on how to enable Modern Authentication can be found here: https://docs.microsoft.com/en-us/exchange/clients-and-mobile-in-exchange-online/enable-or-disable-mo...

For some customers who have disabled Modern Authentication, there is a secondary issue that is causing the client to attempt to use Modern Authentication regardless of the setting. Once the fix has been deployed, the client and service should use the expected configuration.

Current status: We're closely monitoring progress of the fix deployment which has reached approximately 33 percent of the affected infrastructure. We expect that the deployment will complete within the next 48 hours.

Scope of impact: This issue affects a subset of customers and users who are connecting to the service using Basic Authentication and utilize service-based search or a Focused inbox.

Start time: Tuesday, May 5, 2020, at 4:00 AM UTC

Estimated time to resolve: Based on current progress, we expect deployment of the solution to complete by 11:00 PM UTC on Sunday, May 10, 2020. Customers should experience incremental service restoration as the deployment progresses.

Preliminary root cause: A recent update to the Exchange Online service contains a code issue that is causing repeated credential prompts for basic authentication users.

Next update by: Saturday, May 9, 2020, at 11:00 PM UTC
       PublishedTime: 2020-05-08T16:59:41.103Z
     }
   ]
   PostIncidentDocumentUrl: null
   Severity: Sev2
   StartTime: 2020-05-05T04:00:00Z
   Status: Restoring service
   Title: Multiple credential prompts in the Outlook client
   UserFunctionalImpact:
   Workload: Exchange
   WorkloadDisplayName: Exchange Online
}
Labels (2)
Tags (1)
0 Karma
1 Solution

Ultra Champion
index="o365data" sourcetype="o365:service:message" Id=EX212047 
| spath Messages{} output=Messages
| spath WorkloadDisplayName
| stats values(WorkloadDisplayName) as WorkloadDisplayName by Messages
| spath input=Messages
| eval PublishedTime=strptime(PublishedTime."+0000","%FT%T.%3QZ%z")
| sort - PublishedTime
| table PublishedTime WorkloadDisplayName MessageText
| fieldformat PublishedTime=strftime(PublishedTime,"%FT%T.%3Q")

Thanks @joeybroesky
There is sample logs, so I can make query easily.

View solution in original post

0 Karma

Ultra Champion
index="o365data" sourcetype="o365:service:message" Id=EX212047 
| spath Messages{} output=Messages
| spath WorkloadDisplayName
| stats values(WorkloadDisplayName) as WorkloadDisplayName by Messages
| spath input=Messages
| eval PublishedTime=strptime(PublishedTime."+0000","%FT%T.%3QZ%z")
| sort - PublishedTime
| table PublishedTime WorkloadDisplayName MessageText
| fieldformat PublishedTime=strftime(PublishedTime,"%FT%T.%3Q")

Thanks @joeybroesky
There is sample logs, so I can make query easily.

View solution in original post

0 Karma

Path Finder

This worked perfectly! Thanks again for your help to4kawa!

0 Karma

Path Finder

Just noticed PublishedTime is not displaying properly in the alert email but in Splunk is shows properly.

PublishedTime WorkloadDisplayName MessageText
1589444514.360000 Exchange Online A post-incident report has been published.

0 Karma

Ultra Champion

| fieldformat PublishedTime=strftime(PublishedTime,"%FT%T.%3Q")

| eval PublishedTime=strftime(PublishedTime,"%FT%T.%3Q")

make time text.

0 Karma