Monitoring Splunk

ERROR DistBundleRestHandler - Problem untarring file

clocker_splunk
Splunk Employee
Splunk Employee

Running 5.0.1 on Linux, receiving this error over 500 times a day spread across 34 indexers. Using the splunk service account, I was able to untar the the bundle fine so I do not believe it's permission based. Any suggestions on how to resolve the error? I've posted more warnings around the error below:

splunkd.log.5:02-10-2013 19:40:16.107 +0000 ERROR DistBundleRestHandler - Problem untarring file: /opt/splunk/var/run/searchpeers/82C8F892-6A60-451B-9E7B-012BC192118F-1360524901.bundle
splunkd.log.5:02-10-2013 20:06:22.449 +0000 WARN DistBundleRestHandler - There was a problem renaming: /opt/splunk/var/run/searchpeers/82C8F892-6A60-451B-9E7B-012BC192118F-1360526701.tmp -> /opt/splunk/var/run/searchpeers/82C8F892-6A60-451B-9E7B-012BC192118F-1360526701: Directory not empty
splunkd.log.5:02-10-2013 20:06:22.449 +0000 ERROR DistBundleRestHandler - Problem untarring file: /opt/splunk/var/run/searchpeers/82C8F892-6A60-451B-9E7B-012BC192118F-1360526701.bundle
splunkd.log.5:02-10-2013 20:30:14.185 +0000 WARN DistBundleRestHandler - Removed pre-existing temporary directory for untar: /opt/splunk/var/run/searchpeers/DCC02B2A-C40A-41C1-BD92-434A555088B9-1360352701.tmp
splunkd.log.5:02-10-2013 20:30:14.505 +0000 WARN DistBundleRestHandler - There was a problem renaming: /opt/splunk/var/run/searchpeers/DCC02B2A-C40A-41C1-BD92-434A555088B9-1360352701.tmp -> /opt/splunk/var/run/searchpeers/DCC02B2A-C40A-41C1-BD92-434A555088B9-1360352701: Directory not empty
splunkd.log.5:02-10-2013 20:30:14.505 +0000 ERROR DistBundleRestHandler - Problem untarring file: /opt/splunk/var/run/searchpeers/DCC02B2A-C40A-41C1-BD92-434A555088B9-1360352701.bundle

Tags (1)
1 Solution

ewoo
Splunk Employee
Splunk Employee

Bundle replication failures can happen for a variety of reasons. That being said, the ERROR messages here implicate SPL-60740/SPL-74416 as a possible cause -- Splunk instances in a search head pool, under certain conditions, attempt to perform replications that have already completed in the past. This then causes "collisions" with the pre-existing bundles on the indexer.

The fix for this bug will land in future 5.0.x and 6.0.x maintenance releases.

View solution in original post

esalesapns2
Communicator

I'm having the same problem between my deployment server and index cluster. If I remove all the files and directories starting with my deployment server's name from $SPLUNK_HOME/var/run/searchpeers it recovers with no need to restart,

Splunk 7.0.3 (build fa31da744b51).

0 Karma

krish3
Contributor

I am facing the same issue on 6.3.2 😞

Any work around?

0 Karma

rphillips_splk
Splunk Employee
Splunk Employee

encountered these same error messages "Problem untarring file: " and "Directory not empty" in a 6.1.3 release:

SPLUNK VERSION:
VERSION=6.1.3
BUILD=220630
PRODUCT=splunk
PLATFORM=Linux-x86_64

http://answers.splunk.com/answers/221340/warn-streamedsearch-could-not-find-bundles-for-sea.html#ans...

michaeloleary
Path Finder

I too am seeing this issue, however I don't see this on the known issues list in 5.0.4

0 Karma

ewoo
Splunk Employee
Splunk Employee

Bundle replication failures can happen for a variety of reasons. That being said, the ERROR messages here implicate SPL-60740/SPL-74416 as a possible cause -- Splunk instances in a search head pool, under certain conditions, attempt to perform replications that have already completed in the past. This then causes "collisions" with the pre-existing bundles on the indexer.

The fix for this bug will land in future 5.0.x and 6.0.x maintenance releases.

clocker_splunk
Splunk Employee
Splunk Employee

This is a verified bug SPL-62238. The fix will be resolved in v5.0.4.

ctux
Path Finder

mmmm...
Waiting mode: on
:)

0 Karma

ewoo
Splunk Employee
Splunk Employee

The fix is going to land in future 5.0.x and 6.0.x maintenance releases.

0 Karma

adylent
Path Finder

I'm seeing this same issue on 6.0. Did you ever resolve it?

ewoo
Splunk Employee
Splunk Employee

The fix is going to land in a future 5.0.x maintenance release. It didn't really land in 5.0.4.

0 Karma

sdwilkerson
Contributor

Ping! Any updates?

0 Karma

sdwilkerson
Contributor

I'm seeing still seeing the same issue with 5.0.4 as well. Any updates?

0 Karma

ctux
Path Finder

I'm experiencing the same issue.
I did upgrade to version 5.0.4 but the problem persists...

How to check status of bug SPL-62238?

0 Karma

lmyrefelt
Builder

when can we expect to be able to download 5.0.4 ? 🙂

0 Karma

rmcdougal
Path Finder

Did you ever figure out the solution to this issue? I am experiencing the same issues myself.

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to November Tech Talks, Office Hours, and Webinars!

🍂 Fall into November with a fresh lineup of Community Office Hours, Tech Talks, and Webinars we’ve ...

Transform your security operations with Splunk Enterprise Security

Hi Splunk Community, Splunk Platform has set a great foundation for your security operations. With the ...

Splunk Admins and App Developers | Earn a $35 gift card!

Splunk, in collaboration with ESG (Enterprise Strategy Group) by TechTarget, is excited to announce a ...