Splunk Dev

Running script to parse data and output csv

kooixiuhong
New Member

Hi everyone, I am fairly new to Splunk and I have question about scripts. Currently I have a folder with some custom log files that is monitored by Splunk. I have a Python scripts that uses the Splunk SDK and use the rex command to extract fields from the logs and then outputcsv for every log in the folder. The csv files are also monitored by Splunk. I want the script to be ran at specific time each day/week. I am aware Splunk allows scripts scheduling. Does that work for scripts like this or is it better to schedule the script to run on the OS as oppose to on Splunk?

Thanks

0 Karma
1 Solution

renjith_nair
Legend

Hi @kooixiuhong,

You could do it as a scripted input to put all pieces together in splunk itself as mentioned in https://docs.splunk.com/Documentation/Splunk/7.1.1/AdvancedDev/ScriptedInputsIntro

Detailed example : https://docs.splunk.com/Documentation/Splunk/7.1.1/AdvancedDev/ScriptSetup

---
What goes around comes around. If it helps, hit it with Karma 🙂

View solution in original post

0 Karma

woodcock
Esteemed Legend

If you do this as a scripted input, you can still keep the | outputcsv foo as the last line and it will both create the csv and index the results.

0 Karma

renjith_nair
Legend

Hi @kooixiuhong,

You could do it as a scripted input to put all pieces together in splunk itself as mentioned in https://docs.splunk.com/Documentation/Splunk/7.1.1/AdvancedDev/ScriptedInputsIntro

Detailed example : https://docs.splunk.com/Documentation/Splunk/7.1.1/AdvancedDev/ScriptSetup

---
What goes around comes around. If it helps, hit it with Karma 🙂
0 Karma

kooixiuhong
New Member

HI, from all the documentation I read it's seems most scripts using Splunk's scripted inputs provide some output that would be indexed by Splunk. However my scripts just run a search query on log files and then use outputcsv to create csv files. For example if there is 5 logs in the monitor folder, there is a for loop that run the query for each file and outputs 5 csv.

My question is does a script like that qualifies as scripted input? Sorry if it is confusing.

0 Karma

renjith_nair
Legend

Hi @kooixiuhong , you could either output as csv stream and directly post to splunk or could do as you do now to create a file and post it. Below almost matches with your requirement

https://docs.splunk.com/Documentation/Splunk/7.1.1/AdvancedDev/ScriptedInputsIntro#Use_cases_for_scr...

https://docs.splunk.com/Documentation/Splunk/7.1.1/AdvancedDev/ScriptSetup

---
What goes around comes around. If it helps, hit it with Karma 🙂
0 Karma
Get Updates on the Splunk Community!

Leveraging Detections from the Splunk Threat Research Team & Cisco Talos

  Now On Demand  Stay ahead of today’s evolving threats with the combined power of the Splunk Threat Research ...

New in Splunk Observability Cloud: Automated Archiving for Unused Metrics

Automated Archival is a new capability within Metrics Management; which is a robust usage & cost optimization ...

Calling All Security Pros: Ready to Race Through Boston?

Hey Splunkers, .conf25 is heading to Boston and we’re kicking things off with something bold, competitive, and ...