Getting Data In

Files not reindexing even after deleting the fishbucket

cmeo
Contributor

I have some zip files that I need to reindex after cleaning the target index and refining the props.
I cannot get splunk to re-ingest them no matter what--even after cleaning the fishbucket.
Here is the TailingProcessor state for one of them:

<s:key name="/opt/splunkdata/tmp/cm/2014.11/2014001.zip">
              <s:dict>
                <s:key name="file position">0</s:key>
                <s:key name="file size">10239974</s:key>
                <s:key name="parent">/opt/splunkdata/tmp/cm/2014.11</s:key>
                <s:key name="percent">0.00</s:key>
                <s:key name="type">finished reading</s:key>
              </s:dict>
            </s:key>

While percent is 0, it has finished reading it. As far as I can tell from other Answers, this is not supposed to happen.

Btprobe is no help:

splunk cmd btprobe -d /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db --file /opt/splunkdata/tmp/cm/2014.11/2014001.zip
Using logging configuration at /opt/splunk/etc/log-cmdline.cfg.
 record not found

So can't reset it that way. Then I took the radical step of cleaning the fishbucket..still no joy.

I can't see any way around this; and furthermore it looks like some kind of bug--oneshots aren't working either.

Anyone know a solution? Version is 6.4.2

Thanks!

0 Karma

woodcock
Esteemed Legend

You need to change -d /opt/splunk/var to -d /opt/splunkforwarder/var.

skoelpin
SplunkTrust
SplunkTrust

OP could also modify a character at the top of the file so Splunk will see it as a new file.

0 Karma

woodcock
Esteemed Legend

How are you getting the data in? What search are you using to determine that the data is not in? It does not sound like you needed to clear the fishbucket and it may even be that the data actually is in!

0 Karma

cmeo
Contributor

Nope. I added a file monitor pointing to a specific index and it was still empty after an hour or so. When I tried the same thing in the lab I was able to see reindexed events after a few minutes (takes a while to unpack the zip files and get the stuff in).
Searched the index, no results found, checked Settings>Indexes, event count 0.
However I will be checking it again tomorrow...just in case 🙂

0 Karma

splunkreal
Motivator

Try adding crcSalt = <SOURCE> in inputs.conf and restart splunkforwarder on client machines.

* If this helps, please upvote or accept solution 🙂 *

cmeo
Contributor

No forwarder in play here though we may have to set one up to get around this problem.

This is on the indexer. I pieced together a procedure using btprobe to get the key and reset it. Tried it on my lab system, where it worked.

On the customer system, it didn't. Same version of Splunk and OS linux. The key wasn't being found, even though there was a result from call _internal to get the TailingProcessor state.

Plainly there is some sort of file state being stored somewhere else. What I want is to reindex this zipfile, no questions asked. And since cleaning out the fishbucket should reindex everything, and didn't, I'd dearly like to know what's going on.

0 Karma

sahr
Path Finder

Having this same issue now. Did you ever figure this out?

0 Karma

ddrillic
Ultra Champion

@sahr, you better open a new thread and refer to this one for reference...

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...