topic Re: Filter results with value in realtime in Splunk Search

Filter results with value in realtime

dataisbeautiful — Thu, 14 Mar 2024 17:24:16 GMT

I'm trying to build a query to give real time results for a value, but the is a time delay between the data send and indexed. This means when I do a realtime query for last 60s, I get 20s of data and 40s of blank.

I'd like to load the last 60s of recieved data in realtime, not the data recieved in the last 60s.

Any ideas?

I've tried

index=ind sourcetype=src (type=instrument) | where temperature!="" | timechart span=1s values(temperature)

and

index=ind sourcetype=src (type=instrument) | where temperature!= NULL | timechart span=1s values(temperature)

No luck with either

Re: Filter results with value in realtime

gcusello — Thu, 14 Mar 2024 17:32:25 GMT

Hi @dataisbeautiful,

at first don't use the where condition after the main search, this is a bad practice that make your search slower.

Then, you should analyze why you have a delay: have you sufficient resources in your Indexers and Search Heads?

If you have sufficient resources and If there's a delay in indexing You could eventually try to use, in real time, the 60 seconds frome 70 seconds past and 10 seconds past:

index=ind sourcetype=src (type=instrument) earliest=rt-70s latest=rt-10s temperature!="" | timechart span=1s values(temperature)

Ciao.

Giuseppe

Re: Filter results with value in realtime

dataisbeautiful — Fri, 15 Mar 2024 08:44:57 GMT

@Hi @gcusello

Thanks for the reply.

The delay is outside Splunk, it's not something we can solve unfortunately

I've tried adding

earliest=rt-70s latest=rt-10s

but that returned no results, so I broadend the time to

earliest=rt-300s latest=rt

but this also returned no results.

Inspecting the job, the search ran but found no events

Re: Filter results with value in realtime

gcusello — Fri, 15 Mar 2024 11:04:15 GMT

Hi @dataisbeautiful,

what happens running the search not in real time, with the same time window? have you events?

In general I don't like real time searches because every Splunk search uses a CPU and releases it when finished, but a real time search never finishes, so, if many users use one or more real time searches you could kill your system.

Maybe you could use a scheduled report (running e.g. every 5 minutes) and access it in a dashboard (using loadjob), solving in this way also you issue.

Ciao.

Giuseppe

Re: Filter results with value in realtime

dataisbeautiful — Fri, 15 Mar 2024 12:00:59 GMT

Hi @gcusello

Running not in realtime it works fine. I'm starting to think the realtime search isn't the best solution.

If I set the search time to "all time" and use

| head 60

to get the latest 60 samples it does what I'm after

Re: Filter results with value in realtime

gcusello — Fri, 15 Mar 2024 13:06:34 GMT

Hi @dataisbeautiful,

never use All Time!

choose a correct time range and use it,

Ciao.

Giuseppe