Getting Data In

How to add scripts as data input?

tamduong16
Contributor

This is my first time dealing with script input. I tried to place my python script in both of these directories (C:\Program Files\Splunk\bin\scripts, C:\Program Files\Splunk\etc\apps\search\bin) and it hasn't work. I run in python idle and the scripts actually index data into my splunk enviroment. But when I want to add it as a script input, I don't get anything back. It looks like python script never been run. Did I have it in the right file? I didn't create a splunk app for this. What can I do inorder for it to be run in splunk? Here is my python script. Thanks so much!

0 Karma
1 Solution

niketn
Legend

@tamduong16, yes you need to configure Data Input similar to how you would setup File Monitor, Performance Monitors etc. Splunk would need to know what to read, from where to read and how frequently to read, where to index and setting up source/sourcetype etc. These you would need to configure in inputs.conf either through Splunk Web or CLI. Refer to the documentation: http://docs.splunk.com/Documentation/Splunk/latest/Data/Getdatafromscriptedinputs#Add_a_scripted_inp...

Also read about Writing Reliable scripts documentation, as most of the time scripted inputs have a wrapper script as well as maintain your own last indexed data/recovery/parallel execution etc: https://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/ScriptSetup

Once you have completely tested and made your scripted input robust for your scenario, you may be able to build an Add on using Splunk Add On Builder or move towards creating your Modular Input to Splunk

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"

View solution in original post

niketn
Legend

@tamduong16, yes you need to configure Data Input similar to how you would setup File Monitor, Performance Monitors etc. Splunk would need to know what to read, from where to read and how frequently to read, where to index and setting up source/sourcetype etc. These you would need to configure in inputs.conf either through Splunk Web or CLI. Refer to the documentation: http://docs.splunk.com/Documentation/Splunk/latest/Data/Getdatafromscriptedinputs#Add_a_scripted_inp...

Also read about Writing Reliable scripts documentation, as most of the time scripted inputs have a wrapper script as well as maintain your own last indexed data/recovery/parallel execution etc: https://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/ScriptSetup

Once you have completely tested and made your scripted input robust for your scenario, you may be able to build an Add on using Splunk Add On Builder or move towards creating your Modular Input to Splunk

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"

anissabnk
Path Finder

Hello,

I hope you feel good. I have a little question which deals with "Getting Data in" to Splunk with an url.  I understood that I have to write a python scipt for that, but I don't know how to do that, and where I have to include  thois script.

Thank you for your answer.

Have a good day

0 Karma

Elsurion
Communicator

It seems to me, that you don't have enabled the script itself. Have you configured an inputs.conf in the local folder of your search App?
And have you an entry in the inputs.conf like this? (modified for windows support):

splunk@monsterine ✓  09:53 $ cat inputs.conf
[script://.\bin\cfgpull_srv.py]
disabled = false
index = main
interval = 60
source = cfgpull_srv.py
sourcetype = cfgpull_srv_stdout

logloganathan
Motivator

i have mentioned alternative as you are facing issue.

0 Karma

Elsurion
Communicator

what if he likes it the splunk way?

0 Karma

tamduong16
Contributor

@Elsurion I think that's what I'm missing. Since I don't know anything about this. All I did was placing the file inside of the search/bin and expecting it to run on it own. How can I do this? What do I have to do in my input.conf file? I'm very new to this. Thanks for the help

0 Karma

logloganathan
Motivator

Alternative way is
You can get the output of script and save that in the log file without splunk
Then input the log file generated to splunk
Which is very easy way

0 Karma

niketn
Legend

@logloganathan, setting up a scripted input through web is a matter of few clicks and sets up the inputs.conf appropriately.

What you are calling a easy way is actually roundabout. Since after logging data to a file, file monitor input needs to be configured exactly the same way scripted input needs to be setup. Also script will have to be modified to write to a log file as additional change.

Also scripted input capability to ingest data directly to Splunk. Most of the times it is requirement as well as necessity (for example compliance to not to store unencrypted data as flat file). Imagine DB Connect Input or HTTP Event Collector writing to log file and then ingesting to Splunk.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"

niketn
Legend

@tamduong16 if your python script is working on its own and you have configured/scheduled the execution of Scripted input which is enabled, I am surprised as to why nothing is being indexed (success or exception log). However, I would request you to push some checkpoint information from your script to Splunk's _internal index using sys.stderror.write()

sys.stderr.write("Python script is starting up\n")       

Also try to search in internal index for any other metrics captured for your scripted input when you have setup the same.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

tamduong16
Contributor

@niketnilay I'm a little lost. What do you mean by configuring the execution of Scripted input. I thought all I have to do is place the file into search/bin folder and it should run. What could I do to configure this? Thanks!

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...