Getting Data In

Powershell Script to Poll for IIS AppPool consuming large amount of memory- is there a better way?

JHannan
Explorer

Our application developers were looking to poll the service states of their IIS Application Pools.  This would be just like the Windows Service States (Start/Stopped/Disabled).

He wrote a powershell script to check if the Windows Host has IIS installed and if so, checks the service state of all Application Pools.  This was tested in a test environment without any issues.  However, it is now consuming up to 7GB of memory.

My question is, for custom powershell scripts.  I'm not completely versed on Powershell, so can't say for sure if this code is the most optimal method for achieving the results.  Is there a better way?  For such a small task, why would it consume so much memory?

 

 

 

#### SysInternals Process Info #####
Command Line: powershell.exe -command "& {get-content "C:\Windows\TEMP\\inputffdc149fdcf785fb.tmp" | "C:\Program Files\SplunkUniversalFowarder\bin\splunk-powershell.ps1" "C:\Program Files\SplunkUniversalForwarder" ffdc1449fdcf785fb}"

 

 

 

 

The powershell.log file only has two lines when it runs:

 

 

 

07-09-2021 11:11:58.8862744-5 INFO start splunk-powershell.ps1
07-09-2021 11:12:00.5243172-5 INFO launched disposer

 

 

 

The temp file contains info about the app-pool.ps1 stanza

 

 

 

SplunkServerUri:https://127.0.0.1:8089
SplunkSessionKey:<redacted>
stanzas
stanza:App-Pool-State
event_group:-1,1
index:appdevadmin_servers
script:. "$SplunkHome\etc\apps\loves_ta_windows_appdev\bin\powershell\app-pool.ps1"
source:powershell://App-Pool-State
sourcetype:Windows:AppPool

 

 

 

 

Inputs.conf

 

 

 

###### Inputs.conf ######
[powershell://App-Pool-State]
script = . "$SplunkHome\etc\apps\loves_ta_windows_appdev\bin\powershell\app-pool.ps1"
schedule = */5 * * * *
disabled = 0
sourcetype = Windows:AppPool
index = appdevadmin_servers

 

 

 

 

Powershell Script

 

 

 

###### app-pool.ps1 #####
If (Get-WmiObject -Class Win32_ServerFeature -ComputerName $env:computername | Where-Object {$_.name -like "Web Server (IIS)"}){
    Import-Module WebAdministration
    $ApplicationPools = Get-ChildItem IIS:\AppPools;
    Foreach ($ApplicationPool in $ApplicationPools){
      $ApplicationPoolName = $ApplicationPool.Name;
      $ApplicationPoolState = $ApplicationPool.State;
      If ($ApplicationPool.processModel.identityType -eq 'SpecificUser'){
        $UserIdentity = $ApplicationPool.processModel.UserName;
      }
      else {
        $UserIdentity = $ApplicationPool.processModel.identityType;
      }
      Write-Output "
      ApplicationPoolName=`"$ApplicationPoolName`"
      ApplicationPoolState=`"$ApplicationPoolState`"
      UserIdentity=`"$UserIdentity`"
      ";
    }
}

 

 

 

 

0 Karma

dmarling
Builder

Hello,

I believe I know the problem based on some testing we did at my company.  I believe the root issue is when executing powershell with the powershell stanza it is invoking splunk-powershell.exe and that executable is not managing powershells memory at all.  In order to have the memory clear after your script completes you need to put this command at the end of your script:

[System.gc]::Collect()

This will force a garbage collection of powershell.exe's memory. If you do not do this then each iteration of your script will continually add to powershell's memory footprint since the memory is never cleared.  This article was very enlightening in regards to memory management with powershell: https://www.cloudsparkle.be/2021-05-06-PowerShellMemory/

Personally I believe this should be handled by the splunk-powershell.exe process and I believe it should be treated as a defect by splunk.

If this comment/answer was helpful, please up vote it. Thank you.
0 Karma

JasonMcMahan
Explorer

Thank you Dmarling, 

We are adding the line to our script and will be testing, monitoring the memory this week. For us normally it takes approximately 9 days to get memory to the 80-90% range. We really appreciate your help and reply.

Thank you again

JasonMcMahan
Explorer

Good morning, 

We did see some decrease in memory however initially i was not sure if the GC need to be inside or outside the foreach loop.

Originally i had

If (Get-WmiObject -Class Win32_ServerFeature -ComputerName $env:computername | Where-Object {$_.name -like "Web Server (IIS)"}){
    Import-Module WebAdministration
    $ApplicationPools = Get-ChildItem IIS:\AppPools;
    Foreach ($ApplicationPool in $ApplicationPools){
      $ApplicationPoolName = $ApplicationPool.Name;
      $ApplicationPoolState = $ApplicationPool.State;
      If ($ApplicationPool.processModel.identityType -eq 'SpecificUser'){
        $UserIdentity = $ApplicationPool.processModel.UserName;
      }
      else {
        $UserIdentity = $ApplicationPool.processModel.identityType;
      }
      Write-Output "
      ApplicationPoolName=`"$ApplicationPoolName`"
      ApplicationPoolState=`"$ApplicationPoolState`"
      UserIdentity=`"$UserIdentity`"
      ";
    }
    Remove-Module WebAdministration
}
[System.gc]::Collect()

but memory slowly creeped upward so altered the script to 

If (Get-WmiObject -Class Win32_ServerFeature -ComputerName $env:computername | Where-Object {$_.name -like "Web Server (IIS)"}){
    Import-Module WebAdministration
    $ApplicationPools = Get-ChildItem IIS:\AppPools;
    Foreach ($ApplicationPool in $ApplicationPools){
      $ApplicationPoolName = $ApplicationPool.Name;
      $ApplicationPoolState = $ApplicationPool.State;
      If ($ApplicationPool.processModel.identityType -eq 'SpecificUser'){
        $UserIdentity = $ApplicationPool.processModel.UserName;
      }
      else {
        $UserIdentity = $ApplicationPool.processModel.identityType;
      }
      Write-Output "
      ApplicationPoolName=`"$ApplicationPoolName`"
      ApplicationPoolState=`"$ApplicationPoolState`"
      UserIdentity=`"$UserIdentity`"
      ";
      [System.gc]::Collect()
    }
    Remove-Module WebAdministration
}

watching this currently to evaluate. Also testing the following variants.

If (Get-WmiObject -Class Win32_ServerFeature -ComputerName $env:computername | Where-Object {$_.name -like "Web Server (IIS)"}){

    $ApplicationPools = Get-WebConfiguration -Filter '/system.applicationHost/applicationPools/add'
    foreach ($ApplicationPool in $ApplicationPools){
        $ApplicationPoolName = $ApplicationPool.Name;
        $ApplicationPoolState = $ApplicationPool.State;
        If ($ApplicationPool.processModel.identityType -eq 'SpecificUser'){
            $UserIdentity = $ApplicationPool.processModel.UserName;
        }
        else {
            $UserIdentity = $ApplicationPool.processModel.identityType;
        }
        Write-Output "
            ApplicationPoolName: $ApplicationPoolName
            ApplicationPoolState: $ApplicationPoolState
            UserIdentity: $UserIdentity
        ";
        [System.gc]::Collect()
    }
    Remove-Module WebAdministration
}


So watching it a bit longer. Unfortunately this is a slow memory creep 9 days or so before it normally reaches its peek.

0 Karma

JasonMcMahan
Explorer

I wanted to offer and update.

After making the change and performing a restart of the forwarder service memory still climbed pretty quickly to near what it was prior. I then restarted the client servers i running the script on and now 7 days not only as it held between 30-50% i even saw a spike to 76% at one point then it reclaimed 12 hrs later. That at least tells me most likely adding system.gc corrects the issue but requires a restart of the system. I will continue to watch mine for another week or so but consider the matter closed. Thank you

0 Karma

SinghK
Builder

can you ask the powershell  guy to use the Get-IISAppPool cmdlet instead of get-wmiobject and see if he can get the same results.I cannot check this dont have an env to test on. But this should sort the script up.

0 Karma

alexlexxy
Explorer

He did run the cmdlet Get-IISAppPool and got the desired result from power shell prompt. But my issue is how to implement this in splunk to be able to see the same status and use them to set alerts. The AppPool data that is in splunk does not include the status nor event codes associated with the status. So my assessment is that a new input script is required, but can’t figure it out

0 Karma

SinghK
Builder

what i have seenin examples is the cmdlet does not provide eventcodes etc. neither will get-wmiobject. you will have to correlate that yourself.

0 Karma

SinghK
Builder

You can write them to file and the monitor it.

0 Karma

SinghK
Builder

If you can share the output of that cmdlet maybe I can write something with sample data so it gets parsed properly in splunk

0 Karma

alexlexxy
Explorer

I got it working! Thanks to everyone that contributed.

0 Karma

JasonMcMahan
Explorer

What script, cmdlet and output did you end up using?

0 Karma

JasonMcMahan
Explorer

the script works, however it has a memory leak on the servers. That was the issue which opened this as over time memory is slowly eaten.

THe script returns the following to splunk

1/10/22
9:05:01.000 AM
 
ApplicationPoolName="Sites" ApplicationPoolState="Started" UserIdentity="domain\SvcUser"
0 Karma

alexlexxy
Explorer

@JasonMcMahan you created a new HEC input and used the token?

0 Karma

alexlexxy
Explorer

Hi @JHannan I have similar requirement, did you ever get this resolved? If so, could you please share with me the working PS script and the inputs.conf file that you are using. I have a requirement to get the IIS AppPool Status from Splunk which will send an alert if the status changes to "stopped".

Thanks in advance

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...