How to identify IIS Application Pool status changes

[jsmithn]

I know similar questions have been asked a number of times but trying to follow the suggestions given I still cannot get it to work. Perhaps I need to modify the output of the powershell command to create a key/value pair for name+status?

I want the search to show start/stop status changes which we'll use in an actionable alert. Output items should include host, apppool, time of last change, and current status.

The Get-IISAppPool output (ingested into Splunk) looks like this:

[jsmithn]

@to4kawa, thanks for the answer. I modified it slightly just for formatting purposes and to grab the "latest" instead of "last" event. Final query follows:
my_search
| multikv forceheader=1
| table _time host Name Status
| stats latest(_time) as _time latest(Status) as Status dc(Status) as SC by host Name
| where SC > 1
| table _time host Name Status

[alexlexxy]

@jsmithn  I have the same requirement at work, could you please post a walkthrough how you were able to get the fields such as status etc. The logs does not show these fields. Any help will be appreciated. 

[to4kawa]

I see, happy splunking 🙂

[spayneort]

Try adding | ConvertTo-Json -Compress to your PowerShell command.

[to4kawa · Accepted solution]

recommend:

your_search
| multikv forceheader=1
| table _time host Name  Status   CLR_ver  Pipeline_Mode  Start_Mode
| stats last(_time) as _time last(Status) as current_Status dc(Status) as Status_change by Name host
| where Status_change > 1

If all field is extracted, maybe works.

---

previous answer:

| makeresults 
| eval _raw="
Name                Status   CLR_ver  Pipeline_Mode  Start_Mode
----                ------   -------  -------------  ----------
DefaultAppPool      Started  V4.0     Integrated     OnDemand
.NET V4.5 Classic   Started  V4.0     Classic        OnDemand
.NET v4.5           Started  V4.0     Integrated     OnDemand
Monkey              Started  V4.0     Integrated     OnDemand"
`comment("this is your sample")`
`comment("from here, the logic")`
| multikv forceheader=1
| table Name                Status   CLR_ver  Pipeline_Mode  Start_Mode
| stats dc(Status) as Status_change by Name
| where Status_change > 1

Hi, how about this?
Perhaps I need to modify the output of the powershell command to create a key/value pair for name+status?
No, you should provide sample TEXT log.

[jsmithn]

Your solution does identify situations where dc(Status) > 1, so it correctly finds situations when the service state has changed over a given period. But how would I still display the host, time of last change, and current status?

I can add "host" to my display parameter stats dc(Status) AS Status_change BY host,Name but if I add the other columns the dc count becomes invalid.

[to4kawa]

how would I still display the host, time of last change, and current status?
There is no sample text log.
host? time? I can't see them.

[jsmithn]

Aren't host and _time native parameters splunk includes as part of every event ingested? There is no logfile; the powershell script calls the Get-IISAppPool object and the output of that is directly ingested, which is the output I originally included.

I've modified your suggestion like this but it's kludgy:

host=xyz sourcetype=IISAppPool earliest=-5m | multikv forceheader=1
| table host _time Name Status
| append
[ search host=xyz sourcetype=IISAppPool earliest=1h| multikv forceheader=1
| stats dc(Status) AS SC BY host,Name
| where SC > 1 ]
| table host _time Name Status

It shows all the AppPools on a given host when I really only want to display the status of the AppPools where SC > 1. Ideally instead of showing the impacted AppPool twice, it would just show once, and include the time of the last change and the current status.

[to4kawa]

my latest answer is checked?