Splunk Synthetic App: Why am I unable to capture data from my synthetic transaction with a batch file or PowerShell script?

[rmyerspin]

I've been having some trouble getting Splunk to capture data from my synthetic transaction. I believe I have everything installed on the local and remote (forwarder). I started out by creating a .bat file (below) and used the data inputs in the UI to trigger the batch file.

set PATH="C:\Program Files\Python35";%PATH%
set PYTHONPATH=%PYTHONPATH%;"C:\Program Files\Python35\Lib"
set CLASSPATH="C:\Program Files\Java\selenium-server-standalone-2.53.0.jar"
set SELENIUM_SERVER_JAR="C:\Program Files\Java\selenium-server-standalone-2.53.0.jar"

REM Check to see if Java is already running
SETLOCAL EnableExtensions
set EXE=java.exe
FOR /F %%x IN ('tasklist /NH /FI "IMAGENAME eq %EXE%"') DO IF %%x == %EXE% goto RUNNING
echo Not running
goto START

REM if Java is NOT already running start the selenium WebDriver
:START
start java -jar %SELENIUM_SERVER_JAR%

REM Calls the python script transaction
:RUNNING

python "C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk-app-synthetic\bin\PAR_EXTA.py"

exit

This seemed to work sporadically. If I performed a simple index=synthetic it would sometimes return my code as seen below....

Whereas other times it would index the transaction_start and _end events and then I could see the data in the dashboards, so perhaps I'm not calling the Selenium Server properly in my batch file. NOTE: I am able to run the batch file locally and it works without any errors.

Since I was having issues with the batch file I decided to try using a PowerShell script instead. Again I can run the PS script locally and it works properly. However, once I add it through the Data Inputs in Splunk Web, it never seems to trigger, i.e. I'm not seeing any data indexed, not even echoing the code, as previously seen with the batch file. I tried changing .ps1 file permissions to Everyone>Full and also changed the PS executionpolicy to "Unrestricted" w/o success.

# Set Variables
$env:Path = "C:\Program Files\Python35";$env:Path
$env:PYTHONPATH = "C:\Program Files\Python35\Lib"
$env:CLASSPATH = "C:\Program Files\Java\selenium-server-standalone-2.53.0.jar"
$SELENIUM_SERVER_JAR = "C:\Program Files\Java\selenium-server-standalone-2.53.0.jar"
$Synthetic_Script = "C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk-app-synthetic\bin\PAR_EXTA.py"

#Start Selenium Server in the background
Start-Job -ScriptBlock { & java -jar $SELENIUM_SERVER_JAR >console.out 2>console.err}

#Call Python Script
python $Synthetic_Script
Start-Sleep -Seconds 10

#Remove background job
Remove-Job -State Completed

exit

Hopefully someone can point me in the right direction, I'd love to get this app working.

[rmyerspin]

So...after reading a blog regarding the use of PowerShell scripts as scripted inputs it basically said I would need to use a .bat file to call the .ps1 scripts which seems counter productive, so I've ditched PowerShell and am sticking with .bat files to call my Python scripts.

I also figured out why my .bat script is being indexed, because I neglected to put, "@echo off" at the beginning of my .bat file. Now when the transaction runs I only see the transaction_start and transaction_end being indexed. However I have still not figured out why my data is not consistently indexed. Recently its been behaving better, but everyone once in awhile it will stop collecting. I found that by removing the "@echo off" from my .bat file is a good way to see if my script is actually being triggered by the Splunk scripted inputs and that seems to be triggering every time, so now I'm thinking its the Selenium Standalone Server that's not communicating consistently.