Scripted input stream mode

[matthewparry]

Hi,

I've been looking at the documentation i.e http://docs.splunk.com/Documentation/Splunk/4.3.2/Developer/ScriptedInputsIntro but it is vague on actually setting up a streaming scripted input, i.e a script that never exists and continuously sends data to STDOUT.

I have tried setting interval=0 but this has no effect and data will only get set to Splunk when the script is killed or exits.

Any help is much appreciated.

Thanks

[jajskadu]

“php://input allows you to read raw POST data. It is a less memory intensive alternative to $HTTP_RAW_POST_DATA and does not need any special php.ini directives. php://input is not available with enctype=”multipart/form-data”.

[MHibbin · Accepted solution]

Not sure about the " a script that never exists " part. However, I think you want to do is set the following:

interval=-1

This will set a script that runs continuously from Splunk starting. This means that if your script naturally loops itself (e.g. while true ; do ... ; done in Bash), it will continue to run as long as Splunk is.

EDIT: Remember when editing the inputs.conf file, you will need to restart Splunk

Hope this helps

[matthewparry]

Many thanks this fixed the timestamp!

[MHibbin]

If this has answered your question, please mark it as accepted (the empty tick beside the answer), as this will "close" the question, so others are aware.

[MHibbin]

You will need to do something like the following in you props.conf file (create this in the same directory as inputs.conf)

[audit]
TIME_PREFIX = \@timestamp\"\:\"
MAX_TIMESTAMP_LOOK_AHEAD = -1

There are more settings you can play with by looking in the spec file previously shared. You will also need to restart Splunk to apply the changes, this change will only work on new data, not historic.

[MHibbin]

Okay, I would still favour logging to file over syslog, but if you have a solution that's fine!

In terms of the date issue, Splunk will try to find the date by reading from the start of the file until it finds a date it recognises... "Nov" followed by a digit and the time would meet its criteria. You can overide this default action, by configuring your own timestamping.

You will need to edit the props.conf file for this with a little regex. Check out the spec of this file as it is pretty useful.

http://docs.splunk.com/Documentation/Splunk/latest/admin/Propsconf

.... read on below ....

[matthewparry]

Nov 2 11:47:50 host1 /usr/local/bin/audit.rb[32097]: {"@type":"audit","@message":"test message","@source_host":"host1","@source":"audit","@fields":{"action":"check","uniqid":"3aaff7f6d4ab5e4091efdaa93306c2a2","data":"{:process_results=>true}","callerhost":"host1","request_time":1351856868,"agent":"server","caller":"user=test"},"@tags":[],"@timestamp":"2012-11-02T11:47:49.979705Z"}

[matthewparry]

Hi,

I have gone down the route of logging my data to syslog as this seems the only way.

My question now is how to get splunk to use the correct date. Currently the data is coming in from a syslog entry but the data contained (JSON) also contains a timestamp which is what I want to use.

Notice the @timestamp field, this field is not always in the same location in each log entry. I have tested writing this data directly to a file without syslog and Splunk picks up the date correctly, but via syslog, Splunk seems to take preference over the syslog date.

Thanks

[MHibbin]

I would recommend writing the results from the script to a log file instead of to STDOUT. Then have Splunk "monitor" the log file, I've found this a little better, at least then you have some record of what is happening and if something in the script is causing a hang (you could always have two outputs, e.g. one for the STDOUT output and one for actual script logging).

[MHibbin]

Not an expert on ruby scripts (assuming that is what it is). So can't really help with the script, however, I'm going to assume there is some ineffeciency in the script content, that is perhaps not releasing results from memory correctly to STDOUT.

I assume the script runs fine when you test it externally from Splunk?

I know you can test some scripts from Splunk CLI using (example using python):

./splunk cmd python /path/to/file

But I don't think this is available for Ruby.

Also...

[matthewparry]

Hi MHibbin,

I have tried setting this interval, i.e:

[script://$SPLUNK_HOME/etc/apps/search/bin/auduit.rb]
disabled = false
index = audit
interval = -1
source = audit
sourcetype = audit

This doesn't make a difference, I still only receive the events in Splunk once I kill the script or it stops. The script is running a loop and will currently never end unless there is an exception.

Any ideas?