Getting Data In

Can I use REST API, HTTP events, or raw TCP/UDP with scripted input?

yuanliu
SplunkTrust
SplunkTrust

Build scripted inputsGet data from APIs and other remote data interfaces through scripted inputs, etc., point to either streaming (STDOUT) or file monitoring after script runs.  If I am not doing streaming, can I use one of network based methods?  If so, what are merit considerations?

Looking at the interfaces and reading the document, I get the impression that if I do not choose streaming, file monitoring will still have to be set up separately.  Is this correct?  If so, the setting itself is really a high level logic to set up some meta data (plus scheduler).  Instead of writing to local file, a script could also just inject data to one of networking mechanisms.

Labels (2)
0 Karma
1 Solution

venkatasri
SplunkTrust
SplunkTrust

@yuanliu yes in that case one host having UF is enough so you won't duplicate data. You can try scripted input having network connection to forward events. However to work with scripted input you need universal forwarder (UF) other option is without UF is to cron the script.

View solution in original post

0 Karma

venkatasri
SplunkTrust
SplunkTrust

Hi @yuanliu 

If not streaming Network is not an option preferred by Splunk, With script yes we can do many integration without relying on stdout however downsides is if the target system unavailable you will lose the data, establishing connection maintain session to other end is complex (you would have to try batching), other end may throttle then retry shall be applied at source script,  if scripted input restarts/ Splunk UF restarts you may lose the data... these are few demerits that I could think of.

Writing to a file is always best and fail safe which is a kind of store-and-forward model, divides the responsibilities. Scripts triggers on scheduled time and keep writing to a file, Splunk UF on the same host monitors the files and ingest in near real time. All you need is a sufficient disk space, little processing power for UF.

To set-up file monitoring,

----

An upvote would be appreciated if it helps!

yuanliu
SplunkTrust
SplunkTrust

Thanks for the explanation!  Now I remember my previous admin explained store-and-forward advantage to me.  I realize that the subject is related more to distributed sources. (I only did basic file forwarding and host metrics before.  Didn't even remember apps/ directories in forwarder.)

In my case, I can't imagine all the forwarders going after the same data source and forwarding the same data to indexer.  All I wanted is to run a script from one index and "pull" data in.

0 Karma

venkatasri
SplunkTrust
SplunkTrust

@yuanliu yes in that case one host having UF is enough so you won't duplicate data. You can try scripted input having network connection to forward events. However to work with scripted input you need universal forwarder (UF) other option is without UF is to cron the script.

View solution in original post

0 Karma

yuanliu
SplunkTrust
SplunkTrust

need universal forwarder (UF) other option is without UF is to cron the script.

Ah this is the info I missed from the  doco. (A  pointer is appreciated.)  Cron is looking better and better:-)   Thank you again, @venkatasri !

0 Karma

yuanliu
SplunkTrust
SplunkTrust

need universal forwarder (UF) other option is without UF is to cron the script.

Just to clarify: The Web GUI /manager/search/datainputstats lists scripted input (Scripts) in two distinct sections, Local inputs and Forwarded inputs.

Local inputs...
Scripts9Add new
Run custom scripts to collect or generate more data.
Forwarded inputs...
Scripts0Add new
Collect data from scripts installed on forwarders

So, installing a script running on a search heard is an option.  I wish this is easier grasp from the documentation.

0 Karma

venkatasri
SplunkTrust
SplunkTrust

@yuanliu Yes usually scripted inputs  being configured on UF/HF in a distributed set-up, having said that SH having capability of forwarder however to dedicate functions SH is used for ad-hoc searches, knowledge Objects, scheduled searches etc.. less overhead on SH is always good for platform. As long your admin is happy SH can be used as forwarder.

Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!