I have a client that wants to set up a "near" real time search in Splunk. Can this be done (it needs to be continuous), if so, would we need to use a cron job for searches like this or can we dong something like -5rt to rt? Or would that still be considered a "real time" search? The search needs to be run continuously to catch the errors as they come in.
Their search is based off the time selection and it's all time. (index="myindex" REMOTE-DEVICE STATUS CHECK is all there is to it. If the events are greater than one they invoke the script in the $HOME/splunkforwarder/bin/scripts directory. They invoke it when the number of events is greater than zero and if it's true, then they alert and throttle the alerting for 60 minutes.
Anything that ends at "rt" is a real-time search. Ask yourself this: how "continuous" do you want it to be? If you want granularity up to, say, 2 seconds, you can have a bigger base search of -5m to -2s, then add to it with smaller -4s to -2s searches running every 2 seconds.
On forms and dashboards, you can set a refresh time separately on each element or globally at the top level.
It seems like they want to react to something quickly. The question is: how quickly? They must have some "reaction time" allowed.
Another question: what are they using to launch the searches? Some kind of Splunk SDK? If yes, any searches, either one shot or regular, have "earliest_time" and "latest_time" keyword attributes that can be added. I did it in Python but I'd assume it's true in any SDK.
If they need it on a dashboard, many elements can have a refresh value. On the top level (
dashboard tag), there is a
refresh attribute which has a numerical value in seconds. A similar attribute can be in many other elements, such as