Can I use REST API, HTTP events, or raw TCP/UDP with scripted input?

[yuanliu]

Build scripted inputs, Get data from APIs and other remote data interfaces through scripted inputs, etc., point to either streaming (STDOUT) or file monitoring after script runs.  If I am not doing streaming, can I use one of network based methods?  If so, what are merit considerations?

Looking at the interfaces and reading the document, I get the impression that if I do not choose streaming, file monitoring will still have to be set up separately.  Is this correct?  If so, the setting itself is really a high level logic to set up some meta data (plus scheduler).  Instead of writing to local file, a script could also just inject data to one of networking mechanisms.

[venkatasri]

Hi @yuanliu 

If not streaming Network is not an option preferred by Splunk, With script yes we can do many integration without relying on stdout however downsides is if the target system unavailable you will lose the data, establishing connection maintain session to other end is complex (you would have to try batching), other end may throttle then retry shall be applied at source script,  if scripted input restarts/ Splunk UF restarts you may lose the data... these are few demerits that I could think of.

Writing to a file is always best and fail safe which is a kind of store-and-forward model, divides the responsibilities. Scripts triggers on scheduled time and keep writing to a file, Splunk UF on the same host monitors the files and ingest in near real time. All you need is a sufficient disk space, little processing power for UF.

To set-up file monitoring,

• Install UF anyways you need it for scripted input
• configure monitor stanza in inputs.conf - Monitor files and directories with inputs.conf - Splunk Documentation
• copy it to splunk_home/etc/system/local or splunk_home/etc/apps/<app_name>/local
• Restart UF

----

An upvote would be appreciated if it helps!

[yuanliu]

Thanks for the explanation!  Now I remember my previous admin explained store-and-forward advantage to me.  I realize that the subject is related more to distributed sources. (I only did basic file forwarding and host metrics before.  Didn't even remember apps/ directories in forwarder.)

In my case, I can't imagine all the forwarders going after the same data source and forwarding the same data to indexer.  All I wanted is to run a script from one index and "pull" data in.

[venkatasri · Accepted solution]

@yuanliu yes in that case one host having UF is enough so you won't duplicate data. You can try scripted input having network connection to forward events. However to work with scripted input you need universal forwarder (UF) other option is without UF is to cron the script.

[yuanliu]

---

need universal forwarder (UF) other option is without UF is to cron the script.

---

Ah this is the info I missed from the  doco. (A  pointer is appreciated.)  Cron is looking better and better:-)   Thank you again, @venkatasri !

[yuanliu]

---

need universal forwarder (UF) other option is without UF is to cron the script.

---

Just to clarify: The Web GUI /manager/search/datainputstats lists scripted input (Scripts) in two distinct sections, Local inputs and Forwarded inputs.

Local inputs...

Scripts	9	Add new
Run custom scripts to collect or generate more data.

Forwarded inputs...

Scripts	0	Add new
Collect data from scripts installed on forwarders

So, installing a script running on a search heard is an option.  I wish this is easier grasp from the documentation.

[venkatasri]

@yuanliu Yes usually scripted inputs  being configured on UF/HF in a distributed set-up, having said that SH having capability of forwarder however to dedicate functions SH is used for ad-hoc searches, knowledge Objects, scheduled searches etc.. less overhead on SH is always good for platform. As long your admin is happy SH can be used as forwarder.