Running script to parse data and output csv

[kooixiuhong]

Hi everyone, I am fairly new to Splunk and I have question about scripts. Currently I have a folder with some custom log files that is monitored by Splunk. I have a Python scripts that uses the Splunk SDK and use the rex command to extract fields from the logs and then outputcsv for every log in the folder. The csv files are also monitored by Splunk. I want the script to be ran at specific time each day/week. I am aware Splunk allows scripts scheduling. Does that work for scripts like this or is it better to schedule the script to run on the OS as oppose to on Splunk?

Thanks

[woodcock]

If you do this as a scripted input, you can still keep the | outputcsv foo as the last line and it will both create the csv and index the results.

[renjith_nair · Accepted solution]

Hi @kooixiuhong,

You could do it as a scripted input to put all pieces together in splunk itself as mentioned in https://docs.splunk.com/Documentation/Splunk/7.1.1/AdvancedDev/ScriptedInputsIntro

Detailed example : https://docs.splunk.com/Documentation/Splunk/7.1.1/AdvancedDev/ScriptSetup

[kooixiuhong]

HI, from all the documentation I read it's seems most scripts using Splunk's scripted inputs provide some output that would be indexed by Splunk. However my scripts just run a search query on log files and then use outputcsv to create csv files. For example if there is 5 logs in the monitor folder, there is a for loop that run the query for each file and outputs 5 csv.

My question is does a script like that qualifies as scripted input? Sorry if it is confusing.

[renjith_nair]

Hi @kooixiuhong , you could either output as csv stream and directly post to splunk or could do as you do now to create a file and post it. Below almost matches with your requirement

https://docs.splunk.com/Documentation/Splunk/7.1.1/AdvancedDev/ScriptedInputsIntro#Use_cases_for_scr...

https://docs.splunk.com/Documentation/Splunk/7.1.1/AdvancedDev/ScriptSetup