topic Re: Why does my real time search job keeps getting killed? in Splunk Search

Why does my real time search job keeps getting killed?

robertlynch2020 — Thu, 07 Mar 2019 16:38:02 GMT

I have a real time search over the past 5 minutes, however it works for 30 seconds an then it dies.
any ideas?

I have this search at the top of my HOME page users log in and see data flowing into the system from there hosts.

Thanks in advance
Robert 🙂

Re: Why does my real time search job keeps getting killed?

skoelpin — Thu, 07 Mar 2019 16:54:19 GMT

Have you confirmed that your hardware can handle the realtime search? What does the internal logs say when it gets killed?

Re: Why does my real time search job keeps getting killed?

woodcock — Thu, 07 Mar 2019 18:26:38 GMT

When you figure out what is doing that, post back here and let us know. I would like to deploy whatever it is to every Splunk Search Head that I administer!!!!

Re: Why does my real time search job keeps getting killed?

skoelpin — Thu, 07 Mar 2019 18:53:27 GMT

I like your way of thinking.. It's a feature not a bug!

Re: Why does my real time search job keeps getting killed?

chrisyounger — Thu, 07 Mar 2019 22:13:28 GMT

Does it happen even if you leave the tab in the foreground?

These days browsers like Chrome aggressively throttle backgrounded tabs. This can cause the real-time search to think you have closed the window and it will auto-cancel the search. I was able to just replicate this with a lot of tabs open and backgrounding the tab for 30 seconds. There are workarounds you can do to Chrome to prevent this behaviour.

Re: Why does my real time search job keeps getting killed?

robertlynch2020 — Thu, 07 Mar 2019 22:44:43 GMT

So in the end i found the issue.
At the top of my dashboard i had the following refresh="30". When the first refresh comes into play, it killed the real time search i was running. But not all of them, i was running 2, only one stopped.

1st was killed - index=mlc_live | stats count as Events by _time host | timechart span=5s count(Events) as Events by host.
I was running it into a column chart.

2nd did not die (not sure why) index=mlc_live and i am was pursing it out to Events visualization

So i am assuming this is note the way it is suppost to be, but removing the refresh makes it go away.

cheers to all for there suggestions

Rob

Re: Why does my real time search job keeps getting killed?

woodcock — Fri, 08 Mar 2019 02:37:05 GMT

WOW! That is brilliant. Thank you for sharing (this might come up for me later). Be ware that you can set refresh on each panel instead of globally at the top.