Comparing contents of separate but related events

[paulw10]

Hi,

I am trying to find the best way to query events based on windows event log 7036 , around status of a service. 

I want to search for events that contain the words 'stopped' and 'running' and then run a comparison that will trigger an alert if there is an event with "stopped" message but NO message containing running. 

I am trying to create a alert that triggers if a service falls over, but also account for scheduled and approved restarts of the server/service. 

so the search runs for the last 15 mins and if there is an event mentioning stopped, but not one mentioning running for the same service, i want the alert to trigger.

is this possible. 

[rnowitzki]

Hi @paulw10 ,

The event contains the service name and the status, correct?

If both of them are not yet extracted to fields, than this would be the first step. If you need help with that, let us know.

Once you have the fields you can run a search like:

| eventid=7036
| transaction service
| where state="stopped" AND state!="running"

The transaction creates a mv field of the state for each service and with the where you're making sure you find events where there is only "stopped", but not "running" in it.

Now you can set the alert to check every 15 minutes a timeframe of the last 15 minutes (or maybe somtething like -16m to -1m to allow some delay).

Hope this helps.

BR
Ralph
--
Karma and/or Solution tagging appreciated.

[paulw10]

yeah I need help extracting the service and status to fields. have been trying to do this with eval but not getting anywhere

so the eventlog has a message like

Message=The 'Sample Service' service entered the stopped state, or running state

so i want to extract both the service name and stopped/running values into a variable

then perform a comparison, so for a given host check if there is an event with stopped state but no event with running state. 

[rnowitzki]

Hi @paulw10 ,

It would probably make sense to add an extracted field for this information.
Check the documentation how to achieve this.

You can also do with the rex command:

| eventid=7036
| rex field=Message "The\s(?<service>\w+)\sservice\sentered\sthe\s(?<state>\w+)\sstate"
| transaction service
| where state="stopped" AND state!="running"

 
BR
Ralph
--
Karma and/or Solution tagging appreciated.

[paulw10]

that regex is almost what i need but it wont match services with more than 1 word in the name.

how can i do that. ?

[rnowitzki · Accepted solution]

Hi @paulw10 ,

This is now more flexible, it captures everything between (start of the line)"The " and the words "service entered". So it can have multiple strings and also may have numbers or special characters.

| rex field=message "(?<=^The\s)(?<service>.*)\s(?=service\sentered)service\sentered\sthe\s(?<state>\w+)\sstate$"

I was not sure if the message has a dot (.) at the end of the line. If yes, you would have to add it like:

| rex field=message "(?<=^The\s)(?<service>.*)\s(?=service\sentered)service\sentered\sthe\s(?<state>\w+)\sstate\.$"

 
BR
Ralph
--
Karma and/or Solution tagging appreciated.

[paulw10]

Apologies but that's not working for me either.

there are 5 service names i need to track under this alert

McAfee Validation Trust Protection Service

McAfee Task Manager

McAfee McShield

TrueSight Server Automation RSCD Agent

Tanium Client

There is a Dot. at the end of the message 

thanks for your help so far. Regex  really is not my forte 

[rnowitzki]

Hi @paulw10 ,

So the fields service and state are not extracted?  

The RegEx seems to be working. You can check it here: https://regex101.com/r/K1BX9g/1

Also tested in Splunk, you can do a dry run with:

| makeresults 
| eval message="The McAfee Validation Trust Protection Service service entered the running state."
| rex field=message "(?<=^The\s)(?<service>.*)\s(?=service\sentered)service\sentered\sthe\s(?<state>\w+)\sstate\.$"

Can you paste an real life example of the whole message? Maybe I have the wording slightly different or something like that.

BR
Ralph

[paulw10]

thanks all,

i was being a dumb-dumb and had the wrong case on message, it needed to be Message.

this code works great and gives me what I need.

thank you