topic Re: crcSalt is not working in Getting Data In

crcSalt is not working

thisissplunk — Mon, 25 Sep 2017 19:38:17 GMT

I ingested SQL ERRORLOGs and SQLAGENT logs with my forwader but didn't have the props.conf setup correctly. They showed up as binary (hex).

I now have the correct props.conf setup and want to reingest. To do this I set a new crcSalt string in my inputs.conf. However, Splunk still is not reingesting the files.

What gives?

props:

[sql_error]
NO_BINARY_CHECK = true
CHARSET = UTF-16LE

inputs:

[batch:///data/sql/.../ERRORLOG*]
move_policy = sinkhole
index = sql
disabled = false
crcSalt = please_reingest
sourcetype = sql_error

Re: crcSalt is not working

mwdbhyat — Mon, 25 Sep 2017 20:12:29 GMT

You are doing a batch upload - that will destructively upload the file once(delete it from disk after upload)..so that same file shouldnt be there in which case you wouldnt need crcSalt as there is nothing to reingest?

Are you using batch because the files are too big for a file monitor?

Another option to try would be to clean the fishbucket or you could just oneshot the data in again.. here is a link to a few other things you could try:

https://answers.splunk.com/answers/72562/how-to-reindex-data-from-a-forwarder.html

Re: crcSalt is not working

thisissplunk — Mon, 25 Sep 2017 21:41:17 GMT

I checked and the files are there, interestingly enough.

I'm using batch jobs because these are one-shot upload dumps. Nothing else will ever be placed into this index. It's specific data for a specific analysis task.

I can't delete the fishbucket because it would start reingesting everything else in this small splunk instance.

I was trying to figure out how to clean a specific "file" in a fish bucket but I couldn't find instructions.

Re: crcSalt is not working

jkat54 — Tue, 26 Sep 2017 04:25:45 GMT

Two questions:

If your forwarder is "controlled" via a deployment server... does the serverclass require splunkd restart? If not, restart Splunk on the forwarder.
Has move policy always been set to "sinkhole"? If so, then the old data has probably been erased at the source and there's nothing new to ingest.

Re: crcSalt is not working

mwdbhyat — Tue, 26 Sep 2017 08:03:23 GMT

You can use btprobe for a specific file:

https://answers.splunk.com/answers/54147/how-can-i-trigger-the-re-indexing-of-a-single-file.html

Re: crcSalt is not working

jkat54 — Fri, 06 Oct 2017 11:59:05 GMT

@thisissplunk any updates on this?

Re: crcSalt is not working

gjanders — Fri, 06 Oct 2017 12:25:17 GMT

Assuming you did restart the forwarder after tweaking the crcSalt I assume that would trigger re-ingestion, however it might be easier to just force the forwarder to re-index the one file.

The btprobe command can be used to remove an individual entry from the fishbucket, as per this answer , the below is a quote from the linked answer:

splunk cmd btprobe -d $SPLUNK_HOME/var/lib/splunk/fishbucket/splunk_private_db --file $FILE --reset

Also if you can put the file on an indexer or heavy forwarder you could oneshot the file

Re: crcSalt is not working

thisissplunk — Fri, 06 Oct 2017 15:18:37 GMT

No updates. Due to time restraints I just pointed the files to a new index. Next time this happens I'll try the suggestions that people have posted here after I did that.

Re: crcSalt is not working

jkat54 — Fri, 06 Oct 2017 17:23:07 GMT

Ok thanks for the reply. Can you close this question or answer it with your response above and mark it as the answer?

Thanks in advance!

Re: crcSalt is not working

neelshah — Thu, 28 Jun 2018 13:54:28 GMT

Try : https://answers.splunk.com/answers/72562/how-to-reindex-data-from-a-forwarder.html