Splunk Search

How to set up a NEAR real time search in Splunk 6.6.3

nls7010
Path Finder

I have a client that wants to set up a "near" real time search in Splunk. Can this be done (it needs to be continuous), if so, would we need to use a cron job for searches like this or can we dong something like -5rt to rt? Or would that still be considered a "real time" search? The search needs to be run continuously to catch the errors as they come in.

Tags (1)
0 Karma
1 Solution

arkadyz1
Builder

Anything that ends at "rt" is a real-time search. Ask yourself this: how "continuous" do you want it to be? If you want granularity up to, say, 2 seconds, you can have a bigger base search of -5m to -2s, then add to it with smaller -4s to -2s searches running every 2 seconds.

On forms and dashboards, you can set a refresh time separately on each element or globally at the top level.

View solution in original post

0 Karma

nls7010
Path Finder

Their search is based off the time selection and it's all time. (index="myindex" REMOTE-DEVICE STATUS CHECK is all there is to it. If the events are greater than one they invoke the script in the $HOME/splunkforwarder/bin/scripts directory. They invoke it when the number of events is greater than zero and if it's true, then they alert and throttle the alerting for 60 minutes.

0 Karma

arkadyz1
Builder

Anything that ends at "rt" is a real-time search. Ask yourself this: how "continuous" do you want it to be? If you want granularity up to, say, 2 seconds, you can have a bigger base search of -5m to -2s, then add to it with smaller -4s to -2s searches running every 2 seconds.

On forms and dashboards, you can set a refresh time separately on each element or globally at the top level.

0 Karma

nls7010
Path Finder

The clients are looking for a particular phrase in their logs and want it to be a continuous search. Not certain how I would split this up into the two searches you mentioned above.

0 Karma

arkadyz1
Builder

It seems like they want to react to something quickly. The question is: how quickly? They must have some "reaction time" allowed.

Another question: what are they using to launch the searches? Some kind of Splunk SDK? If yes, any searches, either one shot or regular, have "earliest_time" and "latest_time" keyword attributes that can be added. I did it in Python but I'd assume it's true in any SDK.

If they need it on a dashboard, many elements can have a refresh value. On the top level (form or dashboard tag), there is a refresh attribute which has a numerical value in seconds. A similar attribute can be in many other elements, such as table or chart.

0 Karma
Get Updates on the Splunk Community!

Upcoming Webinar: Unmasking Insider Threats with Slunk Enterprise Security’s UEBA

Join us on Wed, Dec 10. at 10AM PST / 1PM EST for a live webinar and demo with Splunk experts! Discover how ...

.conf25 technical session recap of Observability for Gen AI: Monitoring LLM ...

If you’re unfamiliar, .conf is Splunk’s premier event where the Splunk community, customers, partners, and ...

A Season of Skills: New Splunk Courses to Light Up Your Learning Journey

There’s something special about this time of year—maybe it’s the glow of the holidays, maybe it’s the ...