Why am I getting error "'newline' is an invalid keyword argument" using the CLI to run my Python script that writes a CSV file?

[jenniferleenyc]

I'm accessing my python script in $SPLUNK_HOME/bin via command line (in a VM) to see if the code runs correctly. Whenever I run the file outside of commandline, the code works perfectly, but when I run it in the commandline, it gives me this error:

Traceback (most recent call last):
  File "demo.py", line 105, in 
    parse_cert_csv('qualys.csv')
  File "demo.py", line 82, in parse_cert_csv
    with open(FILENAME, 'w', newline='') as csvfile:
TypeError: 'newline' is an invalid keyword argument for this function

My writing-to-file code looks like this:

with open(FILENAME, 'w', newline='') as csvfile:
        keys = list(rows[0].keys()) #keys are the header
        dict_writer = csv.DictWriter(csvfile, fieldnames=keys)
        dict_writer.writeheader()
        dict_writer.writerows(rows)#rows is the data I want to write to the file

Does it matter that my code doesn't work in the command line? And if so, how can I fix it?

[Masa · Accepted solution]

Depending on what you want to do, you should pick how input method Splunk indexes data.

If you're trying to let Splunk schedule the script to run, scripted input or modular input would a good choice. Either way, your script should write data out to standard out where Splunk parse data, instead of writing a file.

For comparison between Modular Input and scripted input, please visit;
https://docs.splunk.com/Documentation/Splunk/6.4.2/AdvancedDev/ModInputsBasicExample

For simple example for Modular Input, please visit and try the examples;
http://dev.splunk.com/view/python-sdk/SP-CAAAER3
http://docs.splunk.com/Documentation/Splunk/6.4.2/AdvancedDev/ModInputsIntro

To run modular input from a command line to validate if the modular input was correctly deployed, please visit the following link;
http://docs.splunk.com/Documentation/Splunk/6.4.2/AdvancedDev/ModInputsDevTools
Here is a copy of a line to run a modular input from cmd. It requires proper scheme and configuration to run it properly. First part is to parse Modular Input configuration from configuration, and pass the output to your Modular Input Script. In this example, twitter.py. In your case, it would be demo.py

splunk cmd splunkd print-modinput-config twitter twitter://SplunkTwitter \
   | splunk cmd python $SPLUNK_HOME/etc/apps/twitter/bin/twitter.py

[jenniferleenyc]

thank you!

[richgalloway]

If your problem is resolved, please accept an answer.

[richgalloway]

How are you running the script on the command line? It should be run in the Splunk environment using $SPLUNK_HOME/bin/splunk cmd python demo.py <args>.

[jenniferleenyc]

Also I moved my python script, demo.py and data file, qualys.csv (for testing purposes) to splunk/home/etc/apps/search/bin (based on this reference: http://docs.splunk.com/Documentation/Splunk/6.4.2/SearchReference/Script) However I'm still confused about the Splunk environment.

[richgalloway]

Is this script something that will be run by Splunk (scripted input, modular input)? If so, it runs in a different environment from your typical shell. The easiest way to run in that environment from a command line is to tell Splunk to do it using the command above.

[jenniferleenyc]

The script will be run by Splunk. Would modular input suffice if I want Splunk to automatically run the data through the script file (which writes the parsed data back into the file) and input that data into Splunk Web?

[richgalloway]

I'm still learning modular inputs, but they're very similar to scripted inputs which will do what you want. When Splunk runs a scripted/modular input, everything the script writes to stdout will be indexed. Of course, when run from the command line, the script's output appears on your screen.

[jenniferleenyc]

thank you!

[jenniferleenyc]

I'm running it through the terminal on a VM.
How do you cd into $SPLUNK_HOME/bin/splunk if splunk is a file?