I'm accessing my python script in $SPLUNK_HOME/bin via command line (in a VM) to see if the code runs correctly. Whenever I run the file outside of commandline, the code works perfectly, but when I run it in the commandline, it gives me this error:
Traceback (most recent call last): File "demo.py", line 105, in parse_cert_csv('qualys.csv') File "demo.py", line 82, in parse_cert_csv with open(FILENAME, 'w', newline='') as csvfile: TypeError: 'newline' is an invalid keyword argument for this function
My writing-to-file code looks like this:
with open(FILENAME, 'w', newline='') as csvfile: keys = list(rows.keys()) #keys are the header dict_writer = csv.DictWriter(csvfile, fieldnames=keys) dict_writer.writeheader() dict_writer.writerows(rows)#rows is the data I want to write to the file
Does it matter that my code doesn't work in the command line? And if so, how can I fix it?
How are you running the script on the command line? It should be run in the Splunk environment using
$SPLUNK_HOME/bin/splunk cmd python demo.py <args>.
I'm running it through the terminal on a VM.
How do you cd into $SPLUNK_HOME/bin/splunk if splunk is a file?
Also I moved my python script, demo.py and data file, qualys.csv (for testing purposes) to splunk/home/etc/apps/search/bin (based on this reference: http://docs.splunk.com/Documentation/Splunk/6.4.2/SearchReference/Script) However I'm still confused about the Splunk environment.
Is this script something that will be run by Splunk (scripted input, modular input)? If so, it runs in a different environment from your typical shell. The easiest way to run in that environment from a command line is to tell Splunk to do it using the command above.
The script will be run by Splunk. Would modular input suffice if I want Splunk to automatically run the data through the script file (which writes the parsed data back into the file) and input that data into Splunk Web?
I'm still learning modular inputs, but they're very similar to scripted inputs which will do what you want. When Splunk runs a scripted/modular input, everything the script writes to stdout will be indexed. Of course, when run from the command line, the script's output appears on your screen.
Depending on what you want to do, you should pick how input method Splunk indexes data.
If you're trying to let Splunk schedule the script to run, scripted input or modular input would a good choice. Either way, your script should write data out to standard out where Splunk parse data, instead of writing a file.
For comparison between Modular Input and scripted input, please visit;
For simple example for Modular Input, please visit and try the examples;
To run modular input from a command line to validate if the modular input was correctly deployed, please visit the following link;
Here is a copy of a line to run a modular input from cmd. It requires proper scheme and configuration to run it properly. First part is to parse Modular Input configuration from configuration, and pass the output to your Modular Input Script. In this example, twitter.py. In your case, it would be demo.py
splunk cmd splunkd print-modinput-config twitter twitter://SplunkTwitter \ | splunk cmd python $SPLUNK_HOME/etc/apps/twitter/bin/twitter.py