Running 6.1.3 on a 2012 AD server which is running the runpowershell.cmd script that contains the following line:
%SystemRoot%\system32\WindowsPowerShell\v1.0\powershell.exe -executionPolicy RemoteSigned -command ". '%SPLUNK_HOME%\etc\apps\%SplunkApp%\bin\powershell\%1'"
Looks like the %1 parameter being passed is ad-repl-stat.ps1.
If left to its own devices, this will consume ALL RAM.
Perusal of logs show nothing.
Exact script running on 20 other identical AD boxes without the excessive disk usage.
Anyone seen this type of behavior before? Any direction on where to look is appreciated.
It'd be nice to debug this to see precisely where the memory is being consumed. But I think the time would be better spent on upgrading to a supported configuration first as @malmoore pointed out, and then see if the problem recurs. For posterity, I'll point to some resources for doing debugging in PowerShell:
If you have v4 or v5 (in preview as of this writing), you can debug remotely which ought to let you attach to the running PowerShell instance that's consuming all of the RAM. I've never done it, and can't promise it would work great if the system is out of memory and not functioning well.
- Remote debugging in the ISE (requires WMF v5 preview)
- Remote debugging from PowerShell console (requires PowerShell v4 on both ends)
For PowerShell v2 and up, which is going to cover practically everything, you can edit a script and set breakpoints, or otherwise start a script in debug mode and do various troubleshooting techniques:
It'd be nice to debug this to see precisely where the memory is being consumed. But I think the time would be better spent on upgrading to a supported configuration first as @malmoore pointed out, and then see if the problem recurs. For posterity, I'll point to some resources for doing debugging in PowerShell:
If you have v4 or v5 (in preview as of this writing), you can debug remotely which ought to let you attach to the running PowerShell instance that's consuming all of the RAM. I've never done it, and can't promise it would work great if the system is out of memory and not functioning well.
- Remote debugging in the ISE (requires WMF v5 preview)
- Remote debugging from PowerShell console (requires PowerShell v4 on both ends)
For PowerShell v2 and up, which is going to cover practically everything, you can edit a script and set breakpoints, or otherwise start a script in debug mode and do various troubleshooting techniques:
I agree, HAL, I will do the upgrade as that needs done anyway. Thank you for the links, I'll keep those handy.
Are you using the TA-DomainController-2012R2 and SA-ModularInput-PowerShell?
Still using TA-DomainController-NT6. Upgrading to the newer TA is on our to-do list.
I was initially told this was consuming disk space but after using Process Explorer, it is actually MEMORY being consumed. After the script runs for 5 minutes it will utilize nearly 3GB of Private Bytes and continue to grab more until the process is killed or the box becomes nearly unusable.
Sorry for that confusion but thanks for the Powershell technique harir9000.
I have changed the tags and title for this question.
I don't think TA-DomainController-NT6 will work with 2012R2. But you say all your other 2012R2 servers run it just fine?
Technically it does function, just does not use the PowerShell modinput ^_^.
Yes, I've got ~20 other ADs in the same Server Class that are all 2012R2. For now I'm just terminating the batch file. I still get AD data - it just bypasses the 3 powershell scripts, ad-health.ps1,ad-repl-stat.ps1 and siteinfo.ps1. I'm guessing I can live with this until I upgrade the TA.
that's "HAL9000" (the AI), with an "R" thrown in for disambiguation. 😉
Was wondering "What the heck does hair9000 mean?". So, now I am enlightened AND off to my optometrist.
Added tag for the windows infrastructure app, which will make your question more visible.
What folder is consuming all of the space? You can use a technique like this to find out: http://blogs.technet.com/b/heyscriptingguy/archive/2011/03/05/use-powershell-to-explore-disk-utiliza...
Thanks hair9000, makes sense!