topic Re: Splunk On Splunk 3.0 generates tons of dispatch files and constantly refreshes Home page in All Apps and Add-ons

Splunk On Splunk 3.0 generates tons of dispatch files and constantly refreshes Home page

mfrost8 — Wed, 05 Jun 2013 15:44:57 GMT

Hello. I seem to be having a couple of problems with Splunk on Splunk 3.0. First, my setup:

Indexer with Splunk 5.0.3 and SOS TA 2.0.4
Splunk search head (5.0.3) configured with search head pooling (although we're currently on only 1 search head) and SOS 3.0

My first problem is that I keep getting these warnings in the UI on the search head that there are too many files in the dispatch directory. I shutdown Splunk yesterday, cleared it out completely and started it up again. As of a few minutes ago there are over 33K directories in my dispatch dir with "sos" in the name. Seems to be mix of names like

scheduler__nobody__sos__RMD5fe2b0603bfc33e11_at_1370430188_7004
subsearch_scheduler__nobody__sos__RMD5fe2b0603bfc33e11_at_1370446101_9051_1370446103.1

The warning I get in the UI coming every few minutes is:

Too many search jobs found in the dispatch directory (found=2689, warning level=2000). This could negatively impact Splunk's performance, consider removing some of the old search jobs.

Too many search jobs found in the dispatch directory (found=2690, warning level=2000). This could negatively impact Splunk's performance, consider removing some of the old search jobs.

Too many search jobs found in the dispatch directory (found=2688, warning level=2000). This could negatively impact Splunk's performance, consider removing some of the old search jobs.

Second problem is that while SOS seems to work, I find that on the default page of the app (seems to be the one labeled "Home - FTR"), it shows me a blank page and then continually refreshes it rapidly. I see this with recent versions of Chrome and Firefox. The other pages in SOS seems to work fine, however.

I've poked around and I haven't noticed anything in the logs that seems to indicate any issue with SOS, but I wouldn't be at all surprised if I missed something.

Thanks

Re: Splunk On Splunk 3.0 generates tons of dispatch files and constantly refreshes Home page

hexx — Mon, 28 Sep 2020 14:02:44 GMT

FINAL UPDATE: Version 3.0.1 of the S.o.S app has been released specifically to address these two issues. If you are running S.o.S on Splunk 5.0.3 and on a pooled search-head, please update to that version at your earliest convenience.

UPDATE: The issue with the S.o.S scheduled searches running a lot more frequently than they are supposed to has been identified as a core Splunk bug (SPL-68970) affecting scheduled searches that do not specify a latest time (dispatch.latest_time in the savedsearches.conf stanza that defines the search) on Splunk 5.0.3 and in a search-head pool.

There are two work-arounds at this time:

Add dispatch.latest_time = now in the savedsearches.conf stanza of the affected searches.
Disable the affected searches. For the S.o.S app, this means disabling the "sos_splunk_instances_info" and "sos_refresh_splunk_servers_cache" scheduled searches.

UPDATE: The issue with the ever-reloading Home view has been identified and fixed (internal bug reference: SUP-720). It is specific to the installation of S.o.S on a pooled search-head and can be circumvented by hitting the http[s]://[splunkweb_host]:[splunkweb_port]/debug/refresh?entity=admin/views endpoint on the search-head.

The fix will be delivered in the next version of S.o.S.

The problem with the unexpectedly high frequency of scheduled searches dispatched by the S.o.S app is still under investigation.

This is a concern for us as it's the second incident of this kind to be reported. Would you please open a support case and attach a diag of the affected instance?

After capturing the diag, I recommend to take the following actions to hopefully address this issue, at least temporarily:

Disable the "sos_refresh_splunk_servers_cache" scheduled search.
In $SPLUNK_HOME/etc/apps/sos/default/data/ui/views, manually overwrite home.xml with home.xml.bak. Reload views by hitting http(s)://(splunkweb_host):(splunkweb_port)/debug/refresh?entity=admin/views

Let me know how this goes as well as your support case number!

Re: Splunk On Splunk 3.0 generates tons of dispatch files and constantly refreshes Home page

mfrost8 — Wed, 05 Jun 2013 17:10:55 GMT

Case #122136. Thanks.

I disabled the scheduled search. Since I have SH pooling on the search head, the path isn't under $SPLUNK_HOME, but I don't think that matters. I only see home.xml and home.xml.FTR in that directory -- no home.xml.bak.

Re: Splunk On Splunk 3.0 generates tons of dispatch files and constantly refreshes Home page

hexx — Mon, 28 Sep 2020 14:02:53 GMT

I took over that case, than you for opening it. I will review the data you provided and make some suggestions as to how we should proceed. I'll update my answer here once we have found the root cause of the issue.
At this point, did you see any improvement after disabling the "sos_refresh_splunk_servers_cache" scheduled search?

Re: Splunk On Splunk 3.0 generates tons of dispatch files and constantly refreshes Home page

hexx — Wed, 05 Jun 2013 19:42:50 GMT

OK, I now have a successful in-house reproduction of the ever-reloading home view issue, which only occurs when search-head pooling is in the picture. Internal bug reference number is SUP-720.

Two work-arounds have been identified so far to get out of this issue:
* Hit the http[s]://[splunkweb_host]:[splunkweb_port]/debug/refresh?entity=admin/views endpoint
* Restart Splunk

Re: Splunk On Splunk 3.0 generates tons of dispatch files and constantly refreshes Home page

christopher_hod — Mon, 28 Sep 2020 14:02:58 GMT

I can confirm that hitting http[s]://[splunkweb_host]:[splunkweb_port]/debug/refresh?entity=admin/views fixed this problem for me

Re: Splunk On Splunk 3.0 generates tons of dispatch files and constantly refreshes Home page

rafrey — Mon, 28 Sep 2020 19:16:42 GMT

I am also having this issue and I was unable to resolve by disabling sos_refresh_splunk_servers_cache and hitting the debug/refresh?entity=admin/views endpoint, however I am using a DS to deploy SOS to my SHP and suspect that whatever the endpoint is changing isn't being persisted because the DS is overwriting the change after the next phone home. Can you tell us what the endpoint is actually doing? I could disable my deployment client and troubleshoot that way, but it would be nice to know what we are actually doing here 🙂

I am using the latest version of sideview and sos (3.2.1) on splunk 6.2.1 using SHP+Mounted Bundles and a DS to deploy sos to the SHP.