Splunk Search

How to set up a NEAR real time search in Splunk 6.6.3

nls7010
Path Finder

I have a client that wants to set up a "near" real time search in Splunk. Can this be done (it needs to be continuous), if so, would we need to use a cron job for searches like this or can we dong something like -5rt to rt? Or would that still be considered a "real time" search? The search needs to be run continuously to catch the errors as they come in.

Tags (1)
0 Karma
1 Solution

arkadyz1
Builder

Anything that ends at "rt" is a real-time search. Ask yourself this: how "continuous" do you want it to be? If you want granularity up to, say, 2 seconds, you can have a bigger base search of -5m to -2s, then add to it with smaller -4s to -2s searches running every 2 seconds.

On forms and dashboards, you can set a refresh time separately on each element or globally at the top level.

View solution in original post

0 Karma

nls7010
Path Finder

Their search is based off the time selection and it's all time. (index="myindex" REMOTE-DEVICE STATUS CHECK is all there is to it. If the events are greater than one they invoke the script in the $HOME/splunkforwarder/bin/scripts directory. They invoke it when the number of events is greater than zero and if it's true, then they alert and throttle the alerting for 60 minutes.

0 Karma

arkadyz1
Builder

Anything that ends at "rt" is a real-time search. Ask yourself this: how "continuous" do you want it to be? If you want granularity up to, say, 2 seconds, you can have a bigger base search of -5m to -2s, then add to it with smaller -4s to -2s searches running every 2 seconds.

On forms and dashboards, you can set a refresh time separately on each element or globally at the top level.

0 Karma

nls7010
Path Finder

The clients are looking for a particular phrase in their logs and want it to be a continuous search. Not certain how I would split this up into the two searches you mentioned above.

0 Karma

arkadyz1
Builder

It seems like they want to react to something quickly. The question is: how quickly? They must have some "reaction time" allowed.

Another question: what are they using to launch the searches? Some kind of Splunk SDK? If yes, any searches, either one shot or regular, have "earliest_time" and "latest_time" keyword attributes that can be added. I did it in Python but I'd assume it's true in any SDK.

If they need it on a dashboard, many elements can have a refresh value. On the top level (form or dashboard tag), there is a refresh attribute which has a numerical value in seconds. A similar attribute can be in many other elements, such as table or chart.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...