Monitoring Splunk

ERROR DistBundleRestHandler - Problem untarring file

clocker_splunk
Splunk Employee
Splunk Employee

Running 5.0.1 on Linux, receiving this error over 500 times a day spread across 34 indexers. Using the splunk service account, I was able to untar the the bundle fine so I do not believe it's permission based. Any suggestions on how to resolve the error? I've posted more warnings around the error below:

splunkd.log.5:02-10-2013 19:40:16.107 +0000 ERROR DistBundleRestHandler - Problem untarring file: /opt/splunk/var/run/searchpeers/82C8F892-6A60-451B-9E7B-012BC192118F-1360524901.bundle
splunkd.log.5:02-10-2013 20:06:22.449 +0000 WARN DistBundleRestHandler - There was a problem renaming: /opt/splunk/var/run/searchpeers/82C8F892-6A60-451B-9E7B-012BC192118F-1360526701.tmp -> /opt/splunk/var/run/searchpeers/82C8F892-6A60-451B-9E7B-012BC192118F-1360526701: Directory not empty
splunkd.log.5:02-10-2013 20:06:22.449 +0000 ERROR DistBundleRestHandler - Problem untarring file: /opt/splunk/var/run/searchpeers/82C8F892-6A60-451B-9E7B-012BC192118F-1360526701.bundle
splunkd.log.5:02-10-2013 20:30:14.185 +0000 WARN DistBundleRestHandler - Removed pre-existing temporary directory for untar: /opt/splunk/var/run/searchpeers/DCC02B2A-C40A-41C1-BD92-434A555088B9-1360352701.tmp
splunkd.log.5:02-10-2013 20:30:14.505 +0000 WARN DistBundleRestHandler - There was a problem renaming: /opt/splunk/var/run/searchpeers/DCC02B2A-C40A-41C1-BD92-434A555088B9-1360352701.tmp -> /opt/splunk/var/run/searchpeers/DCC02B2A-C40A-41C1-BD92-434A555088B9-1360352701: Directory not empty
splunkd.log.5:02-10-2013 20:30:14.505 +0000 ERROR DistBundleRestHandler - Problem untarring file: /opt/splunk/var/run/searchpeers/DCC02B2A-C40A-41C1-BD92-434A555088B9-1360352701.bundle

Tags (1)
1 Solution

ewoo
Splunk Employee
Splunk Employee

Bundle replication failures can happen for a variety of reasons. That being said, the ERROR messages here implicate SPL-60740/SPL-74416 as a possible cause -- Splunk instances in a search head pool, under certain conditions, attempt to perform replications that have already completed in the past. This then causes "collisions" with the pre-existing bundles on the indexer.

The fix for this bug will land in future 5.0.x and 6.0.x maintenance releases.

View solution in original post

esalesapns2
Communicator

I'm having the same problem between my deployment server and index cluster. If I remove all the files and directories starting with my deployment server's name from $SPLUNK_HOME/var/run/searchpeers it recovers with no need to restart,

Splunk 7.0.3 (build fa31da744b51).

0 Karma

krish3
Contributor

I am facing the same issue on 6.3.2 😞

Any work around?

0 Karma

rphillips_splk
Splunk Employee
Splunk Employee

encountered these same error messages "Problem untarring file: " and "Directory not empty" in a 6.1.3 release:

SPLUNK VERSION:
VERSION=6.1.3
BUILD=220630
PRODUCT=splunk
PLATFORM=Linux-x86_64

http://answers.splunk.com/answers/221340/warn-streamedsearch-could-not-find-bundles-for-sea.html#ans...

michaeloleary
Path Finder

I too am seeing this issue, however I don't see this on the known issues list in 5.0.4

0 Karma

ewoo
Splunk Employee
Splunk Employee

Bundle replication failures can happen for a variety of reasons. That being said, the ERROR messages here implicate SPL-60740/SPL-74416 as a possible cause -- Splunk instances in a search head pool, under certain conditions, attempt to perform replications that have already completed in the past. This then causes "collisions" with the pre-existing bundles on the indexer.

The fix for this bug will land in future 5.0.x and 6.0.x maintenance releases.

clocker_splunk
Splunk Employee
Splunk Employee

This is a verified bug SPL-62238. The fix will be resolved in v5.0.4.

ctux
Path Finder

mmmm...
Waiting mode: on
:)

0 Karma

ewoo
Splunk Employee
Splunk Employee

The fix is going to land in future 5.0.x and 6.0.x maintenance releases.

0 Karma

adylent
Path Finder

I'm seeing this same issue on 6.0. Did you ever resolve it?

ewoo
Splunk Employee
Splunk Employee

The fix is going to land in a future 5.0.x maintenance release. It didn't really land in 5.0.4.

0 Karma

sdwilkerson
Contributor

Ping! Any updates?

0 Karma

sdwilkerson
Contributor

I'm seeing still seeing the same issue with 5.0.4 as well. Any updates?

0 Karma

ctux
Path Finder

I'm experiencing the same issue.
I did upgrade to version 5.0.4 but the problem persists...

How to check status of bug SPL-62238?

0 Karma

lmyrefelt
Builder

when can we expect to be able to download 5.0.4 ? 🙂

0 Karma

rmcdougal
Path Finder

Did you ever figure out the solution to this issue? I am experiencing the same issues myself.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In December, the Splunk Threat Research Team had 1 release of new security content via the Enterprise Security ...

Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?

(This is the first of a series of 2 blogs). Splunk Enterprise Security is a fantastic tool that offers robust ...

Index This | What are the 12 Days of Splunk-mas?

December 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...