All Apps and Add-ons

Splunk Add-on for Microsoft Powershell: How to troubleshoot why my Powershell script is not working on a heavy forwarder?

Builder

I have a somewhat complex process I'm trying to get working. The synopsis is this: I have a report that generates a list of machines Splunk has not heard from in at least 12 hours. This report runs on the Search Head, a linux server. That report is piped to a CSV file using outputcsv. I then have a Heavy Forwarder running on a Windows server. On the HF, I wrote a powershell script that retrieves the CSV file, parses the machines, does some powershell 'magic', and then uploads the results to the SH in a new CSV as a lookup table in the Search app. The initial report works fine, and the Powershell script works when I run it from the command line of the HF. However, I want to automate the powershell script and I've been trying to do it in Splunk on the HF using the powershell add-on. I'm currently on 6.2 on all my servers.

So the add-on is installed on the HF and I created an inputs.conf file with the following:

[powershell://check-service]
script = . "c:\Tools\Powershell\test\check_service.ps1"
schedule = 30 */12 * * *
sourcetype = CheckService

Splunk is running on the Windows HF with the same account I'm doing the troubleshooting with. The execution policy for the account is unrestricted.

So I've looked through several of the entries talking about troubleshooting powershell scripts and so my first question is this: I wanted to look at the errors and one of the other entries said to check the powershell logs by running the following search:

index=_internal source="*powershell*.log"

But running this on the SH yields no results. Am I searching for the powershell logs in the wrong place? Are there no logs because I haven't set up the add-on correctly?

I also tried index=_internal source="*powershell*" and index=_internal source="*.ps1" but neither of those searches yielded anything.

Any other suggestions on how to troubleshoot? Any ideas on why this script isn't running?

Thanks.

0 Karma
1 Solution

SplunkTrust
SplunkTrust

Have you tried running those searches locally on the heavy forwarder?
Simply enable the GUI and give that a go.

If that search works fine then you might need to enable the _internal log forwarding to your indexers in order to have it searchable from your Search Heads. Take a look at this answer.

In any case, I think your problem is with the script line. Take a look at the following block from the inputs.conf documentation. I think Splunk expects the script to be in a different directory:

[script://<cmd>]
* Runs <cmd> at a configured interval (see below) and indexes the output.  
* The <cmd> must reside in one of:
  * $SPLUNK_HOME/etc/system/bin/
  * $SPLUNK_HOME/etc/apps/$YOUR_APP/bin/
  * $SPLUNK_HOME/bin/scripts/
* Script path can be an absolute path, make use of an environment variable such
  as $SPLUNK_HOME, or use the special pattern of an initial '.' as the first
  directory to indicate a location inside the current app.   Note that the '.'
  must be followed by a platform-specific directory separator.
  * For example, on UNIX:
        [script://./bin/my_script.sh]
    Or on Windows:
        [script://.\bin\my_program.exe]
    This '.' pattern is strongly recommended for app developers, and necessary
    for operation in search head pooling environments.

View solution in original post

SplunkTrust
SplunkTrust

Have you tried running those searches locally on the heavy forwarder?
Simply enable the GUI and give that a go.

If that search works fine then you might need to enable the _internal log forwarding to your indexers in order to have it searchable from your Search Heads. Take a look at this answer.

In any case, I think your problem is with the script line. Take a look at the following block from the inputs.conf documentation. I think Splunk expects the script to be in a different directory:

[script://<cmd>]
* Runs <cmd> at a configured interval (see below) and indexes the output.  
* The <cmd> must reside in one of:
  * $SPLUNK_HOME/etc/system/bin/
  * $SPLUNK_HOME/etc/apps/$YOUR_APP/bin/
  * $SPLUNK_HOME/bin/scripts/
* Script path can be an absolute path, make use of an environment variable such
  as $SPLUNK_HOME, or use the special pattern of an initial '.' as the first
  directory to indicate a location inside the current app.   Note that the '.'
  must be followed by a platform-specific directory separator.
  * For example, on UNIX:
        [script://./bin/my_script.sh]
    Or on Windows:
        [script://.\bin\my_program.exe]
    This '.' pattern is strongly recommended for app developers, and necessary
    for operation in search head pooling environments.

View solution in original post

Builder

Sorry, I get it now. So I did run the _internal searches locally on the HF and I did not get any results. I used the link you sent and made the changes and now the internal logs are being forwarded correctly.

I then moved my ps script to $SplunkHome\etc\system\bin on the Windows HF and changed the script line in my inputs.conf to:

script = . "$SplunkHome\etc\system\bin\check_wls_beta2.ps1"

The script now runs, but albeit with some errors. The errors have more to do with creation and access to certain files than with Splunk.

I'm going to mark this as resolved. Thanks.

0 Karma

Builder

I need to ask a basic question, how do I run the powershell script from the search?

I'm looking for an example in answers, but haven't found one yet....

Thanks.

(To reiterate, it does run from the command line of the windows server no problem)

0 Karma

SplunkTrust
SplunkTrust

Sorry if my comment wasn't clear enough, what I meant by this:

Have you tried running those searches locally on the heavy forwarder?
Simply enable the GUI and give that a go.

Is that you should try to run your _internal searches from the heavy forwarder just to make sure you are not missing any configuration that forwards those logs to the indexer. By default your _internal logs might not be forwarded automatically to the indexers.

You can't run powershell from the search GUI.

In any case, did you try what I mentioned about the script path? I think you need to place your file somewhere within your SplunkHome directory to get this running. I don't think Splunk is going to run scripts that are outside these locations:

  • $SPLUNK_HOME/etc/system/bin/
  • $SPLUNK_HOME/etc/apps/$YOUR_APP/bin/
  • $SPLUNK_HOME/bin/scripts/
0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!