topic How to run a python script from a universal forwarder? in Getting Data In

How to run a python script from a universal forwarder?

marksheinbaum — Mon, 22 Aug 2022 16:43:37 GMT

I wrote a screen scraping script on a server running Splunk Forwarder version 8.2.3. The script is in a file and runs find from the linux command line. It was developed with Python 3.6. I added the script to our Universal Forwarder local inputs.conf and I can see the script is scheduled successfully. However, it consistently fails with a message in the splunkd.log. I have no idea why the message references python3.7. It is not installed anywhere on the system 3.6 is installed in /usr/bin/python3.6. I tried changing the server.conf properties in local by adding "python.version = python3" in the [general] section and restarting, but to no avail. Please advise what else I might try. Thanks. in advance.

08-18-2022 00:19:45.525 +0000 ERROR ExecProcessor [3423479 ExecProcessor] - message from "python3.7 /opt/splunk/sjcinf8469vmw15/splunkforwarder-8.2.3/splunkforwarder/bin/scripts/scrapeGmrPage.py" /bin/sh: python3.7: command not found

Re: running a python script from a universal forwarder

chaker — Thu, 18 Aug 2022 03:03:49 GMT

If you use ./splunk cmd /pathToScript/scrapeGmrPage.py does it give the same error or different?

Splunk ships with its own python3 parser in its splunk/bin directory, and should be using that to run scripts if it's being run by Splunk.

You may have an environment variable problem with $SPLUNK_HOME or there could be an absolute path somewhere.

Re: running a python script from a universal forwarder

marksheinbaum — Thu, 18 Aug 2022 03:10:31 GMT

The script runs fine when I run as you suggested.

./splunk cmd ~/sjcinf8469vmw15/splunkforwarder/bin/scripts/scrapeGmrPage.py

BTW, I do not see any python interpreters in the splunk bin directory. Not sure if that is relevant:

pwd
/opt/splunk/sjcinf8469vmw15/splunkforwarder/bin

ls
2to3-3.7 copyright.txt genRootCA.sh openssl pip3 prigreypng priweavepng slim wheel
btool easy_install genSignedServerCert.sh pid_check.sh pip3.7 pripalpng pydoc3 splunk
btprobe easy_install-2.7 genWebCert.sh pip pip3.7x pripamtopng pydoc3.7 splunkd
bzip2 easy_install-3.7 idle3 pip2 prichunkpng pripnglsch scripts splunkmon
classify fill_test idle3.7 pip2.7 priforgepng pripngtopam setSplunkEnv srm

Re: running a python script from a universal forwarder

chaker — Thu, 18 Aug 2022 03:22:59 GMT

My apologies, I gave you bad info about the python3 with Splunk UF. See below link

https://community.splunk.com/t5/Splunk-Enterprise/Run-Python-scripts-with-universal-forwarder-to-modify-files-on/m-p/524235/highlight/true#M3772

One work around could be to run the script on a cronjob, have it write STDOUT to a file, and use a file montior to capture the output into Splunk.

Re: running a python script from a universal forwarder

marksheinbaum — Mon, 22 Aug 2022 16:16:05 GMT

I answered this previously. Using the splunk cmd with the script and path works fine. Running the script automatically by configuring the input.conf file in ../etc/apps/SplunkUniversalForwarder/local/ results in the error

08-22-2022 16:14:15.264 +0000 ERROR ExecProcessor [3793782 ExecProcessor] - message from "python3.7 /opt/splunk/sjcinf8469vmw15/splunkforwarder-8.2.3/splunkforwarder/bin/scripts/scrapeGmrPage.py" /bin/sh: python3.7: command not found

Re: running a python script from a universal forwarder

chaker — Tue, 23 Aug 2022 01:37:30 GMT

- If you want Splunk to run your script, put it in a shell script and call the systems /usr/bin/python. Then you can use the script input on the UF to capture the output.

- Alternatively, you can run the script on the systems cron scheduler, and output the contents to a file, then have Splunk monitor that file.