Getting Data In

Why am I getting error "'newline' is an invalid keyword argument" using the CLI to run my Python script that writes a CSV file?

jenniferleenyc
Engager

I'm accessing my python script in $SPLUNK_HOME/bin via command line (in a VM) to see if the code runs correctly. Whenever I run the file outside of commandline, the code works perfectly, but when I run it in the commandline, it gives me this error:

Traceback (most recent call last):
  File "demo.py", line 105, in 
    parse_cert_csv('qualys.csv')
  File "demo.py", line 82, in parse_cert_csv
    with open(FILENAME, 'w', newline='') as csvfile:
TypeError: 'newline' is an invalid keyword argument for this function

My writing-to-file code looks like this:

with open(FILENAME, 'w', newline='') as csvfile:
        keys = list(rows[0].keys()) #keys are the header
        dict_writer = csv.DictWriter(csvfile, fieldnames=keys)
        dict_writer.writeheader()
        dict_writer.writerows(rows)#rows is the data I want to write to the file

Does it matter that my code doesn't work in the command line? And if so, how can I fix it?

0 Karma
1 Solution

Masa
Splunk Employee
Splunk Employee

Depending on what you want to do, you should pick how input method Splunk indexes data.

If you're trying to let Splunk schedule the script to run, scripted input or modular input would a good choice. Either way, your script should write data out to standard out where Splunk parse data, instead of writing a file.

For comparison between Modular Input and scripted input, please visit;
https://docs.splunk.com/Documentation/Splunk/6.4.2/AdvancedDev/ModInputsBasicExample

For simple example for Modular Input, please visit and try the examples;
http://dev.splunk.com/view/python-sdk/SP-CAAAER3
http://docs.splunk.com/Documentation/Splunk/6.4.2/AdvancedDev/ModInputsIntro

To run modular input from a command line to validate if the modular input was correctly deployed, please visit the following link;
http://docs.splunk.com/Documentation/Splunk/6.4.2/AdvancedDev/ModInputsDevTools
Here is a copy of a line to run a modular input from cmd. It requires proper scheme and configuration to run it properly. First part is to parse Modular Input configuration from configuration, and pass the output to your Modular Input Script. In this example, twitter.py. In your case, it would be demo.py

splunk cmd splunkd print-modinput-config twitter twitter://SplunkTwitter \
   | splunk cmd python $SPLUNK_HOME/etc/apps/twitter/bin/twitter.py

View solution in original post

Masa
Splunk Employee
Splunk Employee

Depending on what you want to do, you should pick how input method Splunk indexes data.

If you're trying to let Splunk schedule the script to run, scripted input or modular input would a good choice. Either way, your script should write data out to standard out where Splunk parse data, instead of writing a file.

For comparison between Modular Input and scripted input, please visit;
https://docs.splunk.com/Documentation/Splunk/6.4.2/AdvancedDev/ModInputsBasicExample

For simple example for Modular Input, please visit and try the examples;
http://dev.splunk.com/view/python-sdk/SP-CAAAER3
http://docs.splunk.com/Documentation/Splunk/6.4.2/AdvancedDev/ModInputsIntro

To run modular input from a command line to validate if the modular input was correctly deployed, please visit the following link;
http://docs.splunk.com/Documentation/Splunk/6.4.2/AdvancedDev/ModInputsDevTools
Here is a copy of a line to run a modular input from cmd. It requires proper scheme and configuration to run it properly. First part is to parse Modular Input configuration from configuration, and pass the output to your Modular Input Script. In this example, twitter.py. In your case, it would be demo.py

splunk cmd splunkd print-modinput-config twitter twitter://SplunkTwitter \
   | splunk cmd python $SPLUNK_HOME/etc/apps/twitter/bin/twitter.py

jenniferleenyc
Engager

thank you!

0 Karma

richgalloway
SplunkTrust
SplunkTrust

If your problem is resolved, please accept an answer.

---
If this reply helps you, Karma would be appreciated.
0 Karma

richgalloway
SplunkTrust
SplunkTrust

How are you running the script on the command line? It should be run in the Splunk environment using $SPLUNK_HOME/bin/splunk cmd python demo.py <args>.

---
If this reply helps you, Karma would be appreciated.

jenniferleenyc
Engager

Also I moved my python script, demo.py and data file, qualys.csv (for testing purposes) to splunk/home/etc/apps/search/bin (based on this reference: http://docs.splunk.com/Documentation/Splunk/6.4.2/SearchReference/Script) However I'm still confused about the Splunk environment.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Is this script something that will be run by Splunk (scripted input, modular input)? If so, it runs in a different environment from your typical shell. The easiest way to run in that environment from a command line is to tell Splunk to do it using the command above.

---
If this reply helps you, Karma would be appreciated.
0 Karma

jenniferleenyc
Engager

The script will be run by Splunk. Would modular input suffice if I want Splunk to automatically run the data through the script file (which writes the parsed data back into the file) and input that data into Splunk Web?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I'm still learning modular inputs, but they're very similar to scripted inputs which will do what you want. When Splunk runs a scripted/modular input, everything the script writes to stdout will be indexed. Of course, when run from the command line, the script's output appears on your screen.

---
If this reply helps you, Karma would be appreciated.
0 Karma

jenniferleenyc
Engager

thank you!

0 Karma

jenniferleenyc
Engager

I'm running it through the terminal on a VM.
How do you cd into $SPLUNK_HOME/bin/splunk if splunk is a file?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...