topic Re: Powershell script consuming memory in All Apps and Add-ons

Powershell script consuming memory

Jeff_Lightly_Sp — Tue, 17 Feb 2015 17:36:42 GMT

Running 6.1.3 on a 2012 AD server which is running the runpowershell.cmd script that contains the following line:

%SystemRoot%\system32\WindowsPowerShell\v1.0\powershell.exe -executionPolicy RemoteSigned -command ". '%SPLUNK_HOME%\etc\apps\%SplunkApp%\bin\powershell\%1'"

Looks like the %1 parameter being passed is ad-repl-stat.ps1.

If left to its own devices, this will consume ALL RAM.

Perusal of logs show nothing.

Exact script running on 20 other identical AD boxes without the excessive disk usage.

Anyone seen this type of behavior before? Any direction on where to look is appreciated.

Re: Powershell script consuming memory

halr9000 — Tue, 17 Feb 2015 17:44:26 GMT

Added tag for the windows infrastructure app, which will make your question more visible.

Re: Powershell script consuming memory

Jeff_Lightly_Sp — Tue, 17 Feb 2015 17:45:07 GMT

Thanks hair9000, makes sense!

Re: Powershell script consuming memory

halr9000 — Tue, 17 Feb 2015 17:48:18 GMT

What folder is consuming all of the space? You can use a technique like this to find out: http://blogs.technet.com/b/heyscriptingguy/archive/2011/03/05/use-powershell-to-explore-disk-utilization-on-your-computer.aspx

Re: Powershell script consuming memory

malmoore — Tue, 17 Feb 2015 20:17:15 GMT

What is it consuming all disk space with?
What files are being generated?
Where are the files ending up?

Are you using the TA-DomainController-2012R2 and SA-ModularInput-PowerShell?

Re: Powershell script consuming memory

Jeff_Lightly_Sp — Tue, 17 Feb 2015 21:03:43 GMT

Still using TA-DomainController-NT6. Upgrading to the newer TA is on our to-do list.

I was initially told this was consuming disk space but after using Process Explorer, it is actually MEMORY being consumed. After the script runs for 5 minutes it will utilize nearly 3GB of Private Bytes and continue to grab more until the process is killed or the box becomes nearly unusable.

Sorry for that confusion but thanks for the Powershell technique harir9000.

I have changed the tags and title for this question.

Re: Powershell script consuming memory

halr9000 — Tue, 17 Feb 2015 21:23:04 GMT

that's "HAL9000" (the AI), with an "R" thrown in for disambiguation. 😉

Re: Powershell script consuming memory

Jeff_Lightly_Sp — Tue, 17 Feb 2015 21:56:44 GMT

Was wondering "What the heck does hair9000 mean?". So, now I am enlightened AND off to my optometrist.

Re: Powershell script consuming memory

malmoore — Tue, 17 Feb 2015 23:43:24 GMT

I don't think TA-DomainController-NT6 will work with 2012R2. But you say all your other 2012R2 servers run it just fine?

Re: Powershell script consuming memory

Jeff_Lightly_Sp — Wed, 18 Feb 2015 13:39:30 GMT

Yes, I've got ~20 other ADs in the same Server Class that are all 2012R2. For now I'm just terminating the batch file. I still get AD data - it just bypasses the 3 powershell scripts, ad-health.ps1,ad-repl-stat.ps1 and siteinfo.ps1. I'm guessing I can live with this until I upgrade the TA.

Re: Powershell script consuming memory

jbernt_splunk — Wed, 18 Feb 2015 16:07:33 GMT

Technically it does function, just does not use the PowerShell modinput ^_^.

Re: Powershell script consuming memory

halr9000 — Fri, 20 Feb 2015 16:02:12 GMT

It'd be nice to debug this to see precisely where the memory is being consumed. But I think the time would be better spent on upgrading to a supported configuration first as @malmoore pointed out, and then see if the problem recurs. For posterity, I'll point to some resources for doing debugging in PowerShell:

If you have v4 or v5 (in preview as of this writing), you can debug remotely which ought to let you attach to the running PowerShell instance that's consuming all of the RAM. I've never done it, and can't promise it would work great if the system is out of memory and not functioning well.
- Remote debugging in the ISE (requires WMF v5 preview)
- Remote debugging from PowerShell console (requires PowerShell v4 on both ends)

For PowerShell v2 and up, which is going to cover practically everything, you can edit a script and set breakpoints, or otherwise start a script in debug mode and do various troubleshooting techniques:

How to Debug Scripts in Windows PowerShell ISE

Re: Powershell script consuming memory

Jeff_Lightly_Sp — Mon, 23 Feb 2015 17:20:37 GMT

I agree, HAL, I will do the upgrade as that needs done anyway. Thank you for the links, I'll keep those handy.