I am attempting to use the   "TA-Sysmon-deploy"   Splunkbase app to deploy and maintain Sysmon on our endpoints. I've noticed that the script which checks for sysmon then installs it does run correctly. It always results in a "sysmon not found" situation and re-installs it. This is expected activity if the script does not see sysmon running or it detects is out of date. Nonetheless, the script completes each time by installing sysmon again and again, even thought the host has the proper version of sysmon installed and running. The peculiar thing here is that it works correctly if I run the batch script manually from an Admin (as system) command prompt but not when run by the Splunk Universal Forwarder. I've added an Echo statement so I can the check script variables just before they go into the deployment IF statements. Theyare correct when manually run but are not when executed by Splunk. Any comments or suggestions would be helpful. I have included sample logs and the script below. Thank you, Ken sysmon.log when Splunk runs the batch file via input setting. Thu 09/10/2020- 9:19:40.03 The SplunkUniversalForwarder is installed at C:\Program Files\SplunkUniversalForwarder Thu 09/10/2020- 9:19:40.03 Checking for Sysmon CHECK_SYSMON_VERSION="" CHECK_SYSMON_RUNNIG="" Thu 09/10/2020- 9:19:40.03 Sysmon not found, proceding to install Thu 09/10/2020- 9:19:40.03 Copying the latest config file 0% copied 100% copied 1 file(s) copied. Thu 09/10/2020- 9:19:40.03 Installing Sysmon Thu 09/10/2020- 9:19:40.03 Install complete! sysmon.log when run from and Admin command prompt (as "system") Wed 09/09/2020- 9:08:59.03 The SplunkUniversalForwarder is installed at C:\Program Files\SplunkUniversalForwarder Wed 09/09/2020- 9:08:59.03 Checking for Sysmon CHECK_SYSMON_RUNNIG="1" CHECK_SYSMON_VERSION="1" Wed 09/09/2020- 9:08:59.03 Sysmon found, checking version Wed 09/09/2020- 9:08:59.03 Sysmon already up to date, exiting Here is the script from the deploy.bat file. This batch file is part of "TA-Sysmon-deploy" from Splunkbase. I have added the following to the script while troubleshooting. - SETLOCAL and ENDLOCAL: removes any outside the script variable influences - Enclosed the version check FOR statement in an IF EXIST clause, the script seemed to error out if sysmon.exe did not exist) - added variable output "echo" statements so I can see the variable in the logs just before the IF statements. TA's deploy.bat file     ECHO OFF SETLOCAL FOR /F "delims=" %%i IN ('wmic service SplunkForwarder get Pathname ^| FINDSTR /m service') DO SET SPLUNKDPATH=%%i SET SPLUNKPATH=%SPLUNKDPATH:~1,-28% >> %WINDIR%\sysmon.log ( ECHO %DATE%-%TIME% The SplunkUniversalForwarder is installed at %SPLUNKPATH% ECHO %DATE%-%TIME% Checking for Sysmon FOR /F "delims=" %%c IN ('sc query "Sysmon" ^| FIND /c "RUNNING"') DO ( SET CHECK_SYSMON_RUNNIG=%%c ) IF EXIST %WINDIR%\sysmon.exe ( FOR /F "delims=" %%b IN ('c:\windows\sysmon.exe ^| FIND /c "System Monitor v11.11"') DO ( SET CHECK_SYSMON_VERSION=%%b ) ) ECHO CHECK_SYSMON_VERSION="%CHECK_SYSMON_VERSION%" ECHO CHECK_SYSMON_RUNNIG="%CHECK_SYSMON_RUNNIG%" if "%CHECK_SYSMON_RUNNIG%" == "1" ( ECHO %DATE%-%TIME% Sysmon found, checking version IF "%CHECK_SYSMON_VERSION%" == "1" ( ECHO %DATE%-%TIME% Sysmon already up to date, exiting ENDLOCAL EXIT ) ELSE ( ECHO %DATE%-%TIME% Sysmon binary is outdated, un-installing IF EXIST %WINDIR%\sysmon.exe ( %WINDIR%\sysmon.exe -u ) ) ) ELSE ( ECHO %DATE%-%TIME% Sysmon not found, proceding to install ECHO %DATE%-%TIME% Copying the latest config file COPY /z /y "%SPLUNKPATH%\etc\apps\TA-Sysmon-deploy\bin\config.xml" "C:\windows\" ECHO %DATE%-%TIME% Installing Sysmon "%SPLUNKPATH%\etc\apps\TA-Sysmon-deploy\bin\sysmon.exe" /accepteula -i c:\windows\config.xml | Find /c "Sysmon installed" 1>NUL ECHO %DATE%-%TIME% Install complete! ENDLOCAL EXIT ) ECHO %DATE%-%TIME% Install failed ) ENDLOCAL ... View more

I have two Heavy Forwarders in our environment running the same configuration and running Splunk v7.0.0 - Load balanced to receive syslog data. I noticed the following warning messages while restarting one of them. The other does not show any errors upon restart. . . Undocumented key used in transforms.conf; stanza='pulse_connectsecure_meeting_started' setting='SOURCE_KEY' key='message' Undocumented key used in transforms.conf; stanza='pulse_connectsecure_meeting_updated' setting='SOURCE_KEY' key='message' Undocumented key used in transforms.conf; stanza='pulse_connectsecure_reason' setting='SOURCE_KEY' key='message' Undocumented key used in transforms.conf; stanza='pulse_connectsecure_role' setting='SOURCE_KEY' key='roles' . . There are several TAs or add-ons showing "Undocumented key" errors, all of which are sourced from Splunkbase and contain the original configurations. Not all transforms are affected but about 80% are. Again, I am not seeing this on our other heavy forwarder, nor are we seeing it on our search heads, which also run the same app/add-ons. A search of Splunk Answers did get me some information about using a [accepted_keys] transforms stanza to clean up any errors. But, I would like to get to the core issue before doing any sort of clean up work. Does anyone know what would cause this particular instance of Splunk to report these keys as "undocumented"? Thanks for your help, Ken ... View more

Is anyone collecting Audit and Activity events from the CCURE 9000 application? The logs are in a SQL DB so I assume using the DBConnect2 app is the way to go. I am interested in any advice on what and how to collect the data. Also, any information on the impact to the application caused by collecting the data. Thanks, Ken ... View more

Hello, We are interested in monitoring our O365 and Azure environments using Splunk. There is a TA out available to collect the data but I didn't see a corresponding APP to install on my search heads to view the incoming data. Any suggestions for an app to use or a suite of apps to use for monitoring those events? Does the Splunk App for Windows Infrastructure handle information from Microsoft Cloud data sources? Thank you, Ken ... View more

I have built a search head cluster in our 6.3 Splunk environment. Distributed Searches, App roll out and general searches all work fine until a specific member of the SHCl is elected captain. What would cause the SHCuster to work fine, but fail when 1 specific host is elected captain? The working status of the shcluster looks like this: Captain: dynamic_captain : 1 elected_captain : Mon Oct 26 18:47:19 2015 id : .....406DD1B10D4F initialized_flag : 1 label : workingSH1.sh maintenance_mode : 0 mgmt_uri : https://workingSH1.sh:8089 min_peers_joined_flag : 1 rolling_restart_flag : 0 service_ready_flag : 1 Members: problemcaptain.sh label : problemcaptain.sh mgmt_uri : https://problemcaptain.sh:8089 mgmt_uri_alias : https://10.1.1.2:8089 status : Up workingSH1.sh label : workingSH1.sh mgmt_uri : https://workingSH1.sh:8089 mgmt_uri_alias : https://10.1.2.1:8089 status : Up workingSH2.sh label : workingSH2.sh mgmt_uri : https://workingSH2.sh:8089 mgmt_uri_alias : https://10.1.3.2:8089 status : Up The system seems to work fine in this state. Reports, alerts and manual search all work fine. Deployment of new apps or changes to existing from the deployer work as expected. The SHCluster fails when the problemcaptain.sh host is elected SHCluster captain. The problemcaptain.sh no longer shows as a member in the status, only showing as the Captain. Captain: dynamic_captain : 1 elected_captain : Mon Oct 26 18:27:32 2015 id : .....406DD1B10D4F initialized_flag : 0 label : problemcaptain.sh maintenance_mode : 0 mgmt_uri : https://problemcaptain.sh:8089 min_peers_joined_flag : 0 rolling_restart_flag : 0 service_ready_flag : 0 Members: workingSH1.sh label : workingSH1.sh mgmt_uri : https://workingSH1.sh:8089 mgmt_uri_alias : https://10.1.2.1:8089 status : Up workingSH2.sh label : workingSH2.sh mgmt_uri : https://workingSH2.sh:8089 mgmt_uri_alias : https://10.1.3.2:8089 status : Up ... View more