I'm trying to build a query to give real time results for a value, but the is a time delay between the data send and indexed. This means when I do a realtime query for last 60s, I get 20s of data and 40s of blank.
I'd like to load the last 60s of recieved data in realtime, not the data recieved in the last 60s.
Any ideas?
I've tried
index=ind sourcetype=src (type=instrument)
| where temperature!=""
| timechart span=1s values(temperature)
and
index=ind sourcetype=src (type=instrument)
| where temperature!= NULL
| timechart span=1s values(temperature)
No luck with either
Hi @dataisbeautiful,
at first don't use the where condition after the main search, this is a bad practice that make your search slower.
Then, you should analyze why you have a delay: have you sufficient resources in your Indexers and Search Heads?
If you have sufficient resources and If there's a delay in indexing You could eventually try to use, in real time, the 60 seconds frome 70 seconds past and 10 seconds past:
index=ind sourcetype=src (type=instrument) earliest=rt-70s latest=rt-10s temperature!=""
| timechart span=1s values(temperature)
Ciao.
Giuseppe
@Hi @gcusello
Thanks for the reply.
The delay is outside Splunk, it's not something we can solve unfortunately
I've tried adding
earliest=rt-70s latest=rt-10s
but that returned no results, so I broadend the time to
earliest=rt-300s latest=rt
but this also returned no results.
Inspecting the job, the search ran but found no events
Hi @dataisbeautiful,
what happens running the search not in real time, with the same time window? have you events?
In general I don't like real time searches because every Splunk search uses a CPU and releases it when finished, but a real time search never finishes, so, if many users use one or more real time searches you could kill your system.
Maybe you could use a scheduled report (running e.g. every 5 minutes) and access it in a dashboard (using loadjob), solving in this way also you issue.
Ciao.
Giuseppe
Hi @gcusello
Running not in realtime it works fine. I'm starting to think the realtime search isn't the best solution.
If I set the search time to "all time" and use
| head 60
to get the latest 60 samples it does what I'm after