Powershell Script to Poll for IIS AppPool consuming large amount of memory- is there a better way?

[JHannan]

Our application developers were looking to poll the service states of their IIS Application Pools.  This would be just like the Windows Service States (Start/Stopped/Disabled).

He wrote a powershell script to check if the Windows Host has IIS installed and if so, checks the service state of all Application Pools.  This was tested in a test environment without any issues.  However, it is now consuming up to 7GB of memory.

My question is, for custom powershell scripts.  I'm not completely versed on Powershell, so can't say for sure if this code is the most optimal method for achieving the results.  Is there a better way?  For such a small task, why would it consume so much memory?

#### SysInternals Process Info #####
Command Line: powershell.exe -command "& {get-content "C:\Windows\TEMP\\inputffdc149fdcf785fb.tmp" | "C:\Program Files\SplunkUniversalFowarder\bin\splunk-powershell.ps1" "C:\Program Files\SplunkUniversalForwarder" ffdc1449fdcf785fb}"

The powershell.log file only has two lines when it runs:

07-09-2021 11:11:58.8862744-5 INFO start splunk-powershell.ps1
07-09-2021 11:12:00.5243172-5 INFO launched disposer

The temp file contains info about the app-pool.ps1 stanza

SplunkServerUri:https://127.0.0.1:8089
SplunkSessionKey:<redacted>
stanzas
stanza:App-Pool-State
event_group:-1,1
index:appdevadmin_servers
script:. "$SplunkHome\etc\apps\loves_ta_windows_appdev\bin\powershell\app-pool.ps1"
source:powershell://App-Pool-State
sourcetype:Windows:AppPool

Inputs.conf

###### Inputs.conf ######
[powershell://App-Pool-State]
script = . "$SplunkHome\etc\apps\loves_ta_windows_appdev\bin\powershell\app-pool.ps1"
schedule = */5 * * * *
disabled = 0
sourcetype = Windows:AppPool
index = appdevadmin_servers

Powershell Script

###### app-pool.ps1 #####
If (Get-WmiObject -Class Win32_ServerFeature -ComputerName $env:computername | Where-Object {$_.name -like "Web Server (IIS)"}){
    Import-Module WebAdministration
    $ApplicationPools = Get-ChildItem IIS:\AppPools;
    Foreach ($ApplicationPool in $ApplicationPools){
      $ApplicationPoolName = $ApplicationPool.Name;
      $ApplicationPoolState = $ApplicationPool.State;
      If ($ApplicationPool.processModel.identityType -eq 'SpecificUser'){
        $UserIdentity = $ApplicationPool.processModel.UserName;
      }
      else {
        $UserIdentity = $ApplicationPool.processModel.identityType;
      }
      Write-Output "
      ApplicationPoolName=`"$ApplicationPoolName`"
      ApplicationPoolState=`"$ApplicationPoolState`"
      UserIdentity=`"$UserIdentity`"
      ";
    }
}

[dmarling]

Hello,

I believe I know the problem based on some testing we did at my company.  I believe the root issue is when executing powershell with the powershell stanza it is invoking splunk-powershell.exe and that executable is not managing powershells memory at all.  In order to have the memory clear after your script completes you need to put this command at the end of your script:

[System.gc]::Collect()

This will force a garbage collection of powershell.exe's memory. If you do not do this then each iteration of your script will continually add to powershell's memory footprint since the memory is never cleared.  This article was very enlightening in regards to memory management with powershell: https://www.cloudsparkle.be/2021-05-06-PowerShellMemory/

Personally I believe this should be handled by the splunk-powershell.exe process and I believe it should be treated as a defect by splunk.

[JasonMcMahan]

Thank you Dmarling, 

We are adding the line to our script and will be testing, monitoring the memory this week. For us normally it takes approximately 9 days to get memory to the 80-90% range. We really appreciate your help and reply.

Thank you again

[JasonMcMahan]

Good morning, 

We did see some decrease in memory however initially i was not sure if the GC need to be inside or outside the foreach loop.

Originally i had

If (Get-WmiObject -Class Win32_ServerFeature -ComputerName $env:computername | Where-Object {$_.name -like "Web Server (IIS)"}){
    Import-Module WebAdministration
    $ApplicationPools = Get-ChildItem IIS:\AppPools;
    Foreach ($ApplicationPool in $ApplicationPools){
      $ApplicationPoolName = $ApplicationPool.Name;
      $ApplicationPoolState = $ApplicationPool.State;
      If ($ApplicationPool.processModel.identityType -eq 'SpecificUser'){
        $UserIdentity = $ApplicationPool.processModel.UserName;
      }
      else {
        $UserIdentity = $ApplicationPool.processModel.identityType;
      }
      Write-Output "
      ApplicationPoolName=`"$ApplicationPoolName`"
      ApplicationPoolState=`"$ApplicationPoolState`"
      UserIdentity=`"$UserIdentity`"
      ";
    }
    Remove-Module WebAdministration
}
[System.gc]::Collect()

but memory slowly creeped upward so altered the script to 

If (Get-WmiObject -Class Win32_ServerFeature -ComputerName $env:computername | Where-Object {$_.name -like "Web Server (IIS)"}){
    Import-Module WebAdministration
    $ApplicationPools = Get-ChildItem IIS:\AppPools;
    Foreach ($ApplicationPool in $ApplicationPools){
      $ApplicationPoolName = $ApplicationPool.Name;
      $ApplicationPoolState = $ApplicationPool.State;
      If ($ApplicationPool.processModel.identityType -eq 'SpecificUser'){
        $UserIdentity = $ApplicationPool.processModel.UserName;
      }
      else {
        $UserIdentity = $ApplicationPool.processModel.identityType;
      }
      Write-Output "
      ApplicationPoolName=`"$ApplicationPoolName`"
      ApplicationPoolState=`"$ApplicationPoolState`"
      UserIdentity=`"$UserIdentity`"
      ";
      [System.gc]::Collect()
    }
    Remove-Module WebAdministration
}

watching this currently to evaluate. Also testing the following variants.

If (Get-WmiObject -Class Win32_ServerFeature -ComputerName $env:computername | Where-Object {$_.name -like "Web Server (IIS)"}){

    $ApplicationPools = Get-WebConfiguration -Filter '/system.applicationHost/applicationPools/add'
    foreach ($ApplicationPool in $ApplicationPools){
        $ApplicationPoolName = $ApplicationPool.Name;
        $ApplicationPoolState = $ApplicationPool.State;
        If ($ApplicationPool.processModel.identityType -eq 'SpecificUser'){
            $UserIdentity = $ApplicationPool.processModel.UserName;
        }
        else {
            $UserIdentity = $ApplicationPool.processModel.identityType;
        }
        Write-Output "
            ApplicationPoolName: $ApplicationPoolName
            ApplicationPoolState: $ApplicationPoolState
            UserIdentity: $UserIdentity
        ";
        [System.gc]::Collect()
    }
    Remove-Module WebAdministration
}

So watching it a bit longer. Unfortunately this is a slow memory creep 9 days or so before it normally reaches its peek.

[JasonMcMahan]

I wanted to offer and update.

After making the change and performing a restart of the forwarder service memory still climbed pretty quickly to near what it was prior. I then restarted the client servers i running the script on and now 7 days not only as it held between 30-50% i even saw a spike to 76% at one point then it reclaimed 12 hrs later. That at least tells me most likely adding system.gc corrects the issue but requires a restart of the system. I will continue to watch mine for another week or so but consider the matter closed. Thank you

[SinghK]

can you ask the powershell  guy to use the Get-IISAppPool cmdlet instead of get-wmiobject and see if he can get the same results.I cannot check this dont have an env to test on. But this should sort the script up.

[alexlexxy]

He did run the cmdlet Get-IISAppPool and got the desired result from power shell prompt. But my issue is how to implement this in splunk to be able to see the same status and use them to set alerts. The AppPool data that is in splunk does not include the status nor event codes associated with the status. So my assessment is that a new input script is required, but can’t figure it out

[SinghK]

what i have seenin examples is the cmdlet does not provide eventcodes etc. neither will get-wmiobject. you will have to correlate that yourself.

[SinghK]

You can write them to file and the monitor it.

[SinghK]

If you can share the output of that cmdlet maybe I can write something with sample data so it gets parsed properly in splunk

[alexlexxy]

I got it working! Thanks to everyone that contributed.

[JasonMcMahan]

What script, cmdlet and output did you end up using?

[JasonMcMahan]

the script works, however it has a memory leak on the servers. That was the issue which opened this as over time memory is slowly eaten.

THe script returns the following to splunk

1/10/22 9:05:01.000 AM

ApplicationPoolName="Sites" ApplicationPoolState="Started" UserIdentity="domain\SvcUser"

[alexlexxy]

@JasonMcMahan you created a new HEC input and used the token?

[alexlexxy]

Hi @JHannan I have similar requirement, did you ever get this resolved? If so, could you please share with me the working PS script and the inputs.conf file that you are using. I have a requirement to get the IIS AppPool Status from Splunk which will send an alert if the status changes to "stopped".

Thanks in advance

[astralsailor]

# Import IIS module
Import-Module WebAdministration

# Get the list of Application Pools
$appPools = Get-ChildItem IIS:\AppPools

# Display the App Pool names and their statuses in key-value pairs
foreach ($appPool in $appPools) {
$status = Get-WebAppPoolState -Name $appPool.name
Write-Output "appPoolName=$($appPool.name), appPoolStatus=$($status.Value)"
}

You can expand with more details inf you like.
You can run by either calling with powershell in your stanza or via cmd and call it with powershell that way.