topic Re: Why does /opt/splunk/var/run/searchpeers fill up? in Getting Data In

Why does /opt/splunk/var/run/searchpeers fill up?

ddrillic — Wed, 22 Nov 2017 18:12:32 GMT

On two indexers /opt/splunk/var/run/searchpeers is at 20 GBs of files with delta files and bundle file. Is it safe to delete them?

Re: Why does /opt/splunk/var/run/searchpeers fill up?

ddrillic — Wed, 22 Nov 2017 18:17:03 GMT

searchpeers folder filling up /opt/splunk..what could be the cause??

speaks about it. Is it safe to delete them?

Re: Why does /opt/splunk/var/run/searchpeers fill up?

rphillips_splk — Wed, 22 Nov 2017 19:52:49 GMT

what version of Splunk are you running?

Re: Why does /opt/splunk/var/run/searchpeers fill up?

somesoni2 — Wed, 22 Nov 2017 20:33:52 GMT

You can. The search heads will resend latest knowledge bundles again. For query performance/impact to users, I would say do it during a maintenance period/off-office hours.

Re: Why does /opt/splunk/var/run/searchpeers fill up?

ddrillic — Wed, 22 Nov 2017 22:14:21 GMT

We just upgraded last night from 6.5.2. to 6.6.2. I wonder whether the upgrade has anything to do with this behavior ..

Re: Why does /opt/splunk/var/run/searchpeers fill up?

ddrillic — Wed, 22 Nov 2017 22:14:45 GMT

Perfect. Thank you.

Re: Why does /opt/splunk/var/run/searchpeers fill up?

rphillips_splk — Wed, 22 Nov 2017 23:10:12 GMT

You may be hitting a known bug which can cause this issue. I would consider upgrading to one of the versions listed below and see if that resolves the issue.

BUG: SPL-140831 "Splunk not cleaning up $SPLUNK_HOME/var/run/searchpeers of .delta files and matching directories whose only non-empty subdirectory has the .index extension"

Fixed in: 6.5.6+, 6.6.3+, 7.0.0+

Re: Why does /opt/splunk/var/run/searchpeers fill up?

ddrillic — Mon, 27 Nov 2017 17:02:30 GMT

Much appreciated - we'll upgrade tonight.

Re: Why does /opt/splunk/var/run/searchpeers fill up?

vinaykata — Wed, 03 Jan 2018 01:19:32 GMT

Could you please elaborate on this topic, even I have the same issue storing a lot of bundle and delta files in searchpeers. And we are running 6.6.3. Is it ok to delete those files from search peers? If not what is the best way to reduce the size. Few of them shows more than 900+Mb in size. This causing this error message

"Bundle Replication: Problem replicating config (bundle) to search peer ' xxxx:8089 ', Reading reply to upload: rv=-2, Receive from=https://xxxx:8089 timed out; exceeded 60sec, as per=distsearch.conf/[replicationSettings]/sendRcvTimeout"

Thank you

Re: Why does /opt/splunk/var/run/searchpeers fill up?

micahkemp — Wed, 03 Jan 2018 04:17:40 GMT

The error indicating timeout while replicating search bundles is likely unrelated to the question asked previously.

In your case a larger bundle size may just be taking longer than Splunk expects to replicate out to the search peers. This may have been recently triggered by the search bundle growing due to a new app or lookup, or even just network slowness.

The bundles themselves are just tar files. You can copy one to a temporary directory and untar it to see which, if any, knowledge objects are unnecessarily large. Lookup files are a likely culprit.

Re: Why does /opt/splunk/var/run/searchpeers fill up?

kamal_jagga — Tue, 23 Oct 2018 22:38:28 GMT

I am facing the same issue in 7.0.2.

I have been doing the following things but this only seems to be temporary solution for a few days.
1. Stop splunkd and delete the bundles from searchpeer directory.
2. Data rebalance from Cluster Master.

Kindly advise, if anything else can be done.

Re: Why does /opt/splunk/var/run/searchpeers fill up?

trav271 — Mon, 14 Jan 2019 20:32:57 GMT

Same. Having this problem and I'm on 7.1 so I'm going to open a ticket with support.

Re: Why does /opt/splunk/var/run/searchpeers fill up?

jcastro20111 — Mon, 26 Aug 2019 17:45:51 GMT

I am seeing it in 7.3.1

Re: Why does /opt/splunk/var/run/searchpeers fill up?

joshualemoine — Mon, 10 Feb 2020 19:17:47 GMT

We're also seeing this issue in Splunk 7.2.1. IDX Cluster size = 41 Nodes with $SPLUNK_HOME FS size of 40GB. the Search Peers dir keeps getting up to 25-30GB which triggers the dispatch dir warning about there being less than X amount of size left on the disk. Frustrating.

Re: Why does /opt/splunk/var/run/searchpeers fill up?

splunkcol — Fri, 28 Aug 2020 17:36:15 GMT

@trav271 What answer do I give you support?

Re: Why does /opt/splunk/var/run/searchpeers fill up?

splunkcol — Tue, 01 Sep 2020 01:10:42 GMT

version 8.0.5

Could someone indicate what they are for or what they do and if they can be deleted or not?