All Apps and Add-ons

Powershell script consuming memory

Jeff_Lightly_Sp
Communicator

Running 6.1.3 on a 2012 AD server which is running the runpowershell.cmd script that contains the following line:

%SystemRoot%\system32\WindowsPowerShell\v1.0\powershell.exe -executionPolicy RemoteSigned -command ". '%SPLUNK_HOME%\etc\apps\%SplunkApp%\bin\powershell\%1'"

Looks like the %1 parameter being passed is ad-repl-stat.ps1.

If left to its own devices, this will consume ALL RAM.

Perusal of logs show nothing.

Exact script running on 20 other identical AD boxes without the excessive disk usage.

Anyone seen this type of behavior before? Any direction on where to look is appreciated.

1 Solution

halr9000
Motivator

It'd be nice to debug this to see precisely where the memory is being consumed. But I think the time would be better spent on upgrading to a supported configuration first as @malmoore pointed out, and then see if the problem recurs. For posterity, I'll point to some resources for doing debugging in PowerShell:

If you have v4 or v5 (in preview as of this writing), you can debug remotely which ought to let you attach to the running PowerShell instance that's consuming all of the RAM. I've never done it, and can't promise it would work great if the system is out of memory and not functioning well.
- Remote debugging in the ISE (requires WMF v5 preview)
- Remote debugging from PowerShell console (requires PowerShell v4 on both ends)

For PowerShell v2 and up, which is going to cover practically everything, you can edit a script and set breakpoints, or otherwise start a script in debug mode and do various troubleshooting techniques:

View solution in original post

halr9000
Motivator

It'd be nice to debug this to see precisely where the memory is being consumed. But I think the time would be better spent on upgrading to a supported configuration first as @malmoore pointed out, and then see if the problem recurs. For posterity, I'll point to some resources for doing debugging in PowerShell:

If you have v4 or v5 (in preview as of this writing), you can debug remotely which ought to let you attach to the running PowerShell instance that's consuming all of the RAM. I've never done it, and can't promise it would work great if the system is out of memory and not functioning well.
- Remote debugging in the ISE (requires WMF v5 preview)
- Remote debugging from PowerShell console (requires PowerShell v4 on both ends)

For PowerShell v2 and up, which is going to cover practically everything, you can edit a script and set breakpoints, or otherwise start a script in debug mode and do various troubleshooting techniques:

Jeff_Lightly_Sp
Communicator

I agree, HAL, I will do the upgrade as that needs done anyway. Thank you for the links, I'll keep those handy.

0 Karma

malmoore
Splunk Employee
Splunk Employee
  • What is it consuming all disk space with?
  • What files are being generated?
  • Where are the files ending up?

Are you using the TA-DomainController-2012R2 and SA-ModularInput-PowerShell?

0 Karma

Jeff_Lightly_Sp
Communicator

Still using TA-DomainController-NT6. Upgrading to the newer TA is on our to-do list.

I was initially told this was consuming disk space but after using Process Explorer, it is actually MEMORY being consumed. After the script runs for 5 minutes it will utilize nearly 3GB of Private Bytes and continue to grab more until the process is killed or the box becomes nearly unusable.

Sorry for that confusion but thanks for the Powershell technique harir9000.

I have changed the tags and title for this question.

0 Karma

malmoore
Splunk Employee
Splunk Employee

I don't think TA-DomainController-NT6 will work with 2012R2. But you say all your other 2012R2 servers run it just fine?

0 Karma

jbernt_splunk
Splunk Employee
Splunk Employee

Technically it does function, just does not use the PowerShell modinput ^_^.

0 Karma

Jeff_Lightly_Sp
Communicator

Yes, I've got ~20 other ADs in the same Server Class that are all 2012R2. For now I'm just terminating the batch file. I still get AD data - it just bypasses the 3 powershell scripts, ad-health.ps1,ad-repl-stat.ps1 and siteinfo.ps1. I'm guessing I can live with this until I upgrade the TA.

0 Karma

halr9000
Motivator

that's "HAL9000" (the AI), with an "R" thrown in for disambiguation. 😉

0 Karma

Jeff_Lightly_Sp
Communicator

Was wondering "What the heck does hair9000 mean?". So, now I am enlightened AND off to my optometrist.

halr9000
Motivator

Added tag for the windows infrastructure app, which will make your question more visible.

halr9000
Motivator

What folder is consuming all of the space? You can use a technique like this to find out: http://blogs.technet.com/b/heyscriptingguy/archive/2011/03/05/use-powershell-to-explore-disk-utiliza...

0 Karma

Jeff_Lightly_Sp
Communicator

Thanks hair9000, makes sense!

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...