Getting Data In

Scripted input stream mode

matthewparry
Path Finder

Hi,

I've been looking at the documentation i.e http://docs.splunk.com/Documentation/Splunk/4.3.2/Developer/ScriptedInputsIntro but it is vague on actually setting up a streaming scripted input, i.e a script that never exists and continuously sends data to STDOUT.

I have tried setting interval=0 but this has no effect and data will only get set to Splunk when the script is killed or exits.

Any help is much appreciated.

Thanks

Tags (2)
0 Karma
1 Solution

MHibbin
Influencer

Not sure about the " a script that never exists " part. However, I think you want to do is set the following:

interval=-1

This will set a script that runs continuously from Splunk starting. This means that if your script naturally loops itself (e.g. while true ; do ... ; done in Bash), it will continue to run as long as Splunk is.

EDIT: Remember when editing the inputs.conf file, you will need to restart Splunk

Hope this helps

View solution in original post

jajskadu
New Member

“php://input allows you to read raw POST data. It is a less memory intensive alternative to $HTTP_RAW_POST_DATA and does not need any special php.ini directives. php://input is not available with enctype=”multipart/form-data”.

0 Karma

MHibbin
Influencer

Not sure about the " a script that never exists " part. However, I think you want to do is set the following:

interval=-1

This will set a script that runs continuously from Splunk starting. This means that if your script naturally loops itself (e.g. while true ; do ... ; done in Bash), it will continue to run as long as Splunk is.

EDIT: Remember when editing the inputs.conf file, you will need to restart Splunk

Hope this helps

matthewparry
Path Finder

Many thanks this fixed the timestamp!

0 Karma

MHibbin
Influencer

If this has answered your question, please mark it as accepted (the empty tick beside the answer), as this will "close" the question, so others are aware.

0 Karma

MHibbin
Influencer

You will need to do something like the following in you props.conf file (create this in the same directory as inputs.conf)

[audit]
TIME_PREFIX = \@timestamp\"\:\"
MAX_TIMESTAMP_LOOK_AHEAD = -1

There are more settings you can play with by looking in the spec file previously shared. You will also need to restart Splunk to apply the changes, this change will only work on new data, not historic.

0 Karma

MHibbin
Influencer

Okay, I would still favour logging to file over syslog, but if you have a solution that's fine!

In terms of the date issue, Splunk will try to find the date by reading from the start of the file until it finds a date it recognises... "Nov" followed by a digit and the time would meet its criteria. You can overide this default action, by configuring your own timestamping.

You will need to edit the props.conf file for this with a little regex. Check out the spec of this file as it is pretty useful.

http://docs.splunk.com/Documentation/Splunk/latest/admin/Propsconf

.... read on below ....

0 Karma

matthewparry
Path Finder

Nov 2 11:47:50 host1 /usr/local/bin/audit.rb[32097]: {"@type":"audit","@message":"test message","@source_host":"host1","@source":"audit","@fields":{"action":"check","uniqid":"3aaff7f6d4ab5e4091efdaa93306c2a2","data":"{:process_results=>true}","callerhost":"host1","request_time":1351856868,"agent":"server","caller":"user=test"},"@tags":[],"@timestamp":"2012-11-02T11:47:49.979705Z"}

0 Karma

matthewparry
Path Finder

Hi,

I have gone down the route of logging my data to syslog as this seems the only way.

My question now is how to get splunk to use the correct date. Currently the data is coming in from a syslog entry but the data contained (JSON) also contains a timestamp which is what I want to use.

Notice the @timestamp field, this field is not always in the same location in each log entry. I have tested writing this data directly to a file without syslog and Splunk picks up the date correctly, but via syslog, Splunk seems to take preference over the syslog date.

Thanks

0 Karma

MHibbin
Influencer

I would recommend writing the results from the script to a log file instead of to STDOUT. Then have Splunk "monitor" the log file, I've found this a little better, at least then you have some record of what is happening and if something in the script is causing a hang (you could always have two outputs, e.g. one for the STDOUT output and one for actual script logging).

0 Karma

MHibbin
Influencer

Not an expert on ruby scripts (assuming that is what it is). So can't really help with the script, however, I'm going to assume there is some ineffeciency in the script content, that is perhaps not releasing results from memory correctly to STDOUT.

I assume the script runs fine when you test it externally from Splunk?

I know you can test some scripts from Splunk CLI using (example using python):

./splunk cmd python /path/to/file

But I don't think this is available for Ruby.

Also...

0 Karma

matthewparry
Path Finder

Hi MHibbin,

I have tried setting this interval, i.e:

[script://$SPLUNK_HOME/etc/apps/search/bin/auduit.rb]
disabled = false
index = audit
interval = -1
source = audit
sourcetype = audit

This doesn't make a difference, I still only receive the events in Splunk once I kill the script or it stops. The script is running a loop and will currently never end unless there is an exception.

Any ideas?

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...