Getting Data In

Files not reindexing even after deleting the fishbucket

cmeo
Contributor

I have some zip files that I need to reindex after cleaning the target index and refining the props.
I cannot get splunk to re-ingest them no matter what--even after cleaning the fishbucket.
Here is the TailingProcessor state for one of them:

<s:key name="/opt/splunkdata/tmp/cm/2014.11/2014001.zip">
              <s:dict>
                <s:key name="file position">0</s:key>
                <s:key name="file size">10239974</s:key>
                <s:key name="parent">/opt/splunkdata/tmp/cm/2014.11</s:key>
                <s:key name="percent">0.00</s:key>
                <s:key name="type">finished reading</s:key>
              </s:dict>
            </s:key>

While percent is 0, it has finished reading it. As far as I can tell from other Answers, this is not supposed to happen.

Btprobe is no help:

splunk cmd btprobe -d /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db --file /opt/splunkdata/tmp/cm/2014.11/2014001.zip
Using logging configuration at /opt/splunk/etc/log-cmdline.cfg.
 record not found

So can't reset it that way. Then I took the radical step of cleaning the fishbucket..still no joy.

I can't see any way around this; and furthermore it looks like some kind of bug--oneshots aren't working either.

Anyone know a solution? Version is 6.4.2

Thanks!

0 Karma

woodcock
Esteemed Legend

You need to change -d /opt/splunk/var to -d /opt/splunkforwarder/var.

skoelpin
SplunkTrust
SplunkTrust

OP could also modify a character at the top of the file so Splunk will see it as a new file.

0 Karma

woodcock
Esteemed Legend

How are you getting the data in? What search are you using to determine that the data is not in? It does not sound like you needed to clear the fishbucket and it may even be that the data actually is in!

0 Karma

cmeo
Contributor

Nope. I added a file monitor pointing to a specific index and it was still empty after an hour or so. When I tried the same thing in the lab I was able to see reindexed events after a few minutes (takes a while to unpack the zip files and get the stuff in).
Searched the index, no results found, checked Settings>Indexes, event count 0.
However I will be checking it again tomorrow...just in case 🙂

0 Karma

realsplunk
Motivator

Try adding crcSalt = SOURCE in inputs.conf and restart splunkforwarder on client machines.

cmeo
Contributor

No forwarder in play here though we may have to set one up to get around this problem.

This is on the indexer. I pieced together a procedure using btprobe to get the key and reset it. Tried it on my lab system, where it worked.

On the customer system, it didn't. Same version of Splunk and OS linux. The key wasn't being found, even though there was a result from call _internal to get the TailingProcessor state.

Plainly there is some sort of file state being stored somewhere else. What I want is to reindex this zipfile, no questions asked. And since cleaning out the fishbucket should reindex everything, and didn't, I'd dearly like to know what's going on.

0 Karma

sahr
Path Finder

Having this same issue now. Did you ever figure this out?

0 Karma

ddrillic
Ultra Champion

@sahr, you better open a new thread and refer to this one for reference...

0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!